Can german citizen with denied student visa in the past fly in to USA without filling out Esta from online?

No, you need to fill it out. You can not enter the united states without a visa or green card or pport.