it'll cause overfitting with no symptoms since evaluation uses the testing set which the model has now seen before. Does this seem like it could have potential?

I'm trying something new and am wondering if those more experienced than me think it's a good idea. Every X epochs without improvement I train my model for Y epochs using the testing data, instead of the training data. This is in the hopes of giving it some understanding that's nonexistent in the training set, but present in the testing set. This is mostly for small(ish) data sets. I'm using this sparingly, so X is large and Y is small. Batch size for the testing set is much larger than for the training set so it gets very generalized averages for gradient updates. Still, I do worry it'll cause overfitting with no symptoms since evaluation uses the testing set which the model has now seen before. Does this seem like it could have potential?