Does it apply any kind of persistence/memory on the instantaneous exogenous variables you feed it? E.g. if you feed it the exogenous variable of "temperature right now", is it able to create a new exogenous feature "average temperature over the last three time steps"?

No, you have to handcraft all (transformations) of exogenous features. But since it's really all linear regression, that's usually reasonably straightforward.