The expression "by Jingo" is apparently a minced oath that appeared rarely in print, but which may be traced as far back as to at least the 17th century in a transparent euphemism for "by Jesus". The OED attests the first appearance in 1694, in an English edition of the works of François Rabelais as a translation for the French par Dieu! ("by God!").
The full expression is "By the living Jingo", substituting for the phrase "By the living God" (referring to the Christian belief that the god-man Jesus Christ rose from the dead and now lives). "Living Jingo" refers to a legend attached to St. Gengulphus (Jingo for short) that after his martyrdom by being hacked to pieces, the pieces animated and hopped out to accuse his murderer. A version of the story appears in the Ingoldsby Legends.
The form "by Gingo!" is also recorded in the 18th century.
The expression "hey Jingo"/"hey Yingo" was also known in the vocabulary of illusionists and jugglers as a cue for magic appearance of objects (cf. "abracadabra"). Martim de Albuquerque in his 1881 "Notes and Queries" mentions a 1679 printed usage of the expression.
I'm digging a hole
In my back yard
I hit rock and sprung water
I dove in
And Ended up in Beijing
Stole me a bike
Joined up with the monks
Because silence
Is worth more these days
Oh Than AOL Time Warner
Or Martha Stewart stocks
If you can get it
Grab as much as you can
And hide it in your basement
For the next disaster
Silence for the next disaster
Oh and my guitar
Sounds so good
Because I can't tell you why
And if distance makes the heart grow fonder
Then oh look at you and I yea
Look at you and I
If you can get it
Grab as much as you can
And hide it in your basement
For the next disaster
A little perfection
During your disaster
So bring me down
To where I can't see
I'll pray for some medicine
Explain to you I'm lost
You said you're not lost you're free
You said you're free baby
You're free
You said you're free
If you can get it
Grab as much as you can
Hide it in your basement
For the next disaster
Yea during your disasters