The  quadratic system to normalize (system (1)):

xdot=x y//Expand
ydot=-y+2 (x^2)+x y//Expand

define lowest degrees of \mathcal{Q}:

deg={{0,1},{1,0},{2,-1}}

define a subset of \mathcal{Q} ordered lexicographically:

ls={};Do[ Do[{u={deg[[i]][[1]],deg[[i]][[2]]+j}; AppendTo[ls,u]},{j,0,2}],{i,1,Length[deg]}]

ls

reduce the set using a  bound on monomials:

lis={}; Do[{Q=ls[[i]];If[(Q[[1]]<3&& Q[[2]]<3 && Q[[1]]+ Q[[2]] <3), AppendTo[lis,Q]]; },{i,1,Length[ls]}]

lis

to get a subset qq of \mathcal{Q}  add {0,0} to lis

qq={{0,0},{0,1},{0,2},{1,0},{1,1},{2,-1},{2,0}}

monomials of the system

mon={{0,0},{0,1},{1,0},{2,-1}}

F={};Do[AppendTo[F,{0,0}],{i,1,Length[qq]}];

F

the coefficients of    system of system (1)

sys={{0,-1},{1,0},{0,1},{0,2}}

the   lattice vector field (F)_qq to be normalized :

Do[ {i=FirstPosition[qq,mon[[j]]],F[[i]]=sys[[j]]},{j,1,Length[mon]}]

F

qq

m=Length[qq]

comd[uu_,vv_]:={vv[[2]].uu[[1]] vv[[1]]-uu[[2]].vv[[1]] uu[[1]],uu[[2]]+vv[[2]]}

adj[k_,hhV_,ff_]:=Module[{res,rr,j,u,s},{ res={};Do[AppendTo[res,{0,0}],{s,1,Length[ff]}];u=First[FirstPosition[qq,hhV[[2]]]]; Do[If[u+i<k+1,{rr=comd[hhV,{ff[[i]],qq[[i]]}];If[MemberQ[qq,rr[[2]]],{j=FirstPosition[qq,rr[[2]]];res[[j]]=rr[[1]];}]}],{i,1,Length[qq]}]};res]

ads[k_,i_,hk_,fj_]:=Module[{ff,s,u}, res={};Do[AppendTo[res,{0,0}],{u,1,Length[fj]}]; ff=fj;s=1;  While[s<i,{ff=adj[k,hk,ff];res=res+(1/((s)!)) ff};s++];res]

to compute the normal form till the term (4,4):

k1=FirstPosition[qq,{2,0}][[1]]

lambda={0,-1}

af=F

set k1+1 as the first argument in ads

Do[{Q=qq[[s]];If[Q.lambda ≠0, { h={ 1/(Q.lambda) af[[s]],qq[[s]]};af=af+ ads[k1+1,m,h,af]}]},{s,1,m}]

af

eq1=0; Do[eq1=eq1+af[[i]][[1]]   x^qq[[i]][[1]]   y^qq[[i]][[2]],{i,1,m}]; eq1=x eq1;

eq2=0; Do[eq2=eq2+af[[i]][[2]]   x^qq[[i]][[1]]   y^qq[[i]][[2]],{i,1,m}]; eq2=y eq2;

eq1 is  the first equation of the normal form,
eq2 is   the second equation of the normal form

eq1

eq2//Expand