PROGRAMMING

DOCUMENTATION for The

emotional stock market

By Max Dovey
12/04/2011

The document is a log book for my process of developing

programs that can stream information from the social website
twitter in real time. It contains experiments, code, dialogue, and
creative thoughts during the devlopment process.

I saw this process in two stages, creating a code in processing

that could communicate with twitter (e.g extract data) and
secondly visualize it in real time.

There are 3 parts that connect for this real time visualization to
work.

Processing is a graphically interface for artists

and designers to create computer language. Creatives wanting to
work with computers directly (no software) have found it the most
intuitive way to design programs. I have been aware of
Processing for a year but have only scratched the surface,
knowing that it was a very powerful tool but afraid of code.

‘Twitter4j is an unofficial library for for the Twitter Api’. This

means it provides ways to take the information from Twitter
(Twitter API) and use it in other ways (programs made in
Processing). It links the commands made in Processing to the
Twitter API.
 Twitter is a social media network and microblogging site that
allows personal updates and instant conversations. It has
become a tool for user generated media and is now an infamous
communication site.

Brief
I want to monitor expressions on twitter that use terms that define
individual emotions. I want to be bale to visually present real time
use of these terms in a graphically way.

1st stage
Getting code that can respond and interact with Twitter

Three Steps to a Tweeting Processing Sketch

1. Set up a Twitter application

2. Turn twitter4j into a Processing library
3. Use the right line of code to authenticate the account

Installation
I began by downloading the Twitter4j library and importing it into
Processing. This was not that easy
My Sketch still refused to work saying that it cannot find the
Twitter4j Library.
I then posted an example code into the sketch and uploaded into
the forum.
hey been using this code to get twitter4j working but keep getting this error
saying unexpected character "\" but this character doesn't exist in the code
so i don't really get it.
it also says:
processing.app.debug.RunnerException: unexpected char: '\'
at processing.app.Sketch.preprocess(Sketch.java:1352)
at processing.app.Sketch.preprocess(Sketch.java:1205)
at processing.app.Sketch.build(Sketch.java:1568)
at processing.app.Sketch.build(Sketch.java:1553)
at processing.app.Editor$DefaultRunHandler.run(Editor.java:1485)
at java.lang.Thread.run(Thread.java:680)

any help would be much appreciated thank you. 

Copy code
1. /*
2. Posts a message to a Twitter account when you press the mouse
button.
3. Uses Twitter4j, http://twitter4j.org.
4. For more info: http://tinkerlondon.com/now/2010/09/13/oauth-
twitter-and-processing/
5. Daniel Soltis, September 2010
6. */
7. import twitter4j.conf.*;
8. import twitter4j.internal.async.*;
9. import twitter4j.internal.org.json.*;
10. import twitter4j.internal.logging.*;
11. import twitter4j.http.*;
12. import twitter4j.api.*;
13. import twitter4j.util.*;
14. import twitter4j.internal.http.*;
15. import twitter4j.*;
16. String msg = “Automatically posted from Processing”;
17. //copy and paste these from your application in
dev.twitter.com
18. String consumer_key =
“publicconsumerkeyfromtwitterdevpage”;
19. String consumer_secret =
“secretconsumerkeyfromtwitterdevpage”;
20. String oauth_token = “accesstokenforyouraccount”;
21. String oauth_token_secret =
“secretaccesstokenforyouraccount”;
22. color bgcolor = color(255);
23. long timer;
24. void setup() {
 25. size(640,480);
26. }
27. void draw() {
28. background(bgcolor);
29. if (millis()-timer > 2000) bgcolor = color(255);
30. }
31. void mousePressed() {
32. Twitter twitter = new
TwitterFactory().getOAuthAuthorizedInstance (
33. consumer_key, consumer_secret,
34. new AccessToken( oauth_token, oauth_token_secret) );
35. try {
36. Status st = twitter.updateStatus(msg + ” ” + second());
37. println(“Successfully updated the status to [" + st.getText()
+ "].”);
38. bgcolor = color(0,0,255);
39. timer = millis();
40. }
41. catch (TwitterException e) {
42. println(e.getStatusCode());
43. }
44. }
  1 user has this question 

Re: Problems with twitter4j

by 
Jesse Scott
 on 12-Apr-2011 03:14 PM
Contributed Library Forum.

Re: Problems with twitter4j

by 
phi.lho
 on 12-Apr-2011 10:29 PM
Looks like a bug in Processing itself, in the pre-processor. What version
of Processing do you use? Maybe you can try and upgrade (or even
downgrade). Or just restart the PDE, somebody with a similar issue had
success with this step...

The geniuses say reboot?

I attempt to re-install different versions of Processing.

14/04/11
processing version 1.2.1 deleted

download and test 1.2 same result

download and test 1 not work.

Seems like the twitter4j library is not getting into processing. No

matter what version of processing I use. I try and install an earlier
version of Twitter4j library
Success
I managed to import the twitter4j library by using an older version -
twitter4j2.1.3 following this guide
http://arduino.cc/forum/index.php/topic,57488.0.html

Connection made

Once I had Successfully imported the Twitter4j library I could run

some simple example sketches made for example –
Tweeting from clicking my mouse

14/04/10
Now I managed to run an app that tweeted when I clicked my
mouse – I got slightly overexcited

License
Before I quickly over-used the Twitter API I had to register before
they blocked me for spamming the site.
This authentication for third party clients (tweeting without twitter)
only came to practice last summer. Its an act of data security on
Twitters part. So I tried to register on the devlopers site for Twitter.
http://dev.twitter.com/

There registration page was down, so I tweeted Twitter –

Twitter Developers then repaired their site and I was able to
register as a developer. This means I can create programs to use
Twitter information in different ways.

Searching
I then slammed in the earlier code that searches twitter.
This works, you can set the amount you retrieve for various search
terms. I searched 100 happy and 100 sad. See here.

Problems with this code is that it wont fetch more than 100 per
query.

how to accurately
I began to encounter my next issue,
portray the amount of emotional
expressions in realtime
Twitter has strict rules and what you do with the data. Below are restrictions
on third party use.
 Anonymous calls are based on the IP of the host and are
permitted 150 requests per hour. This classification includes
unauthenticated requests (such as RSS feeds), and
authenticated requests to resources that do not require
authentication.
 OAuth calls are permitted 350 requests per hour.

That’s 6 a minute
1 every 10 secs

The search API does not require authentication.

I was trying to work out how often I would grab results

from twitter to give an accurate portrait. This goes back to
how often you can Search Twitters API (350 ph). 1 search
every 10 seconds was generating a lot more than 100
expressions using the word Happy or sad.

I worked out that if I can do a new search every 3

Seconds instead of the 100 limit per search.

By asking for the Last ID , I can make sure I don’t count

the same tweets twice. For example every search begins
at the Last ID from the last search and is not limited to 100
results.

By only printing and counting the ID numbers I could save

myself a lot of trouble. By printing the user, the text and
the Id number it was running slow, this way I just count
how many people have used the search term since the
last count. The most efficient way to make a bar graph.
Once I had made a method for counting the tweets for a
specific search term I needed to visualize it.

Then I found ‘The Twitter Party Race’ by Daniel Soltis.

This was made in May 2010 to track tweets related to the

general election.
Daniel uses the last Id to gather new tweets with the
search term every 3 seconds. Below is some of his code
that I have used.

This Code counts each tweet and strings it into its

category. There the amount is turned into a percentage
relative of the other total tweets. If it goes up 1 percent
than the pixel rectangle increases.
To see if it would work I tested my own search terms -
1st Draft.
Obviously want to play around with the layout a lot more.
This is a format that searches and updates every 3
seconds. The percentage helps to put things into a
context.

It does stop after 60,000 tweets.

Problems –
Pretty happy but….
All search terms begin fast and exciting when there is 100
or less tweets. Once you get to 1000 they remain static
due to their size. When analyzing emotional data it seems
to all become static, meaning there is constant
expressions of basic emotional categories on Twitter. I
can use more search terms in one label to widen my
search, but perhaps all subjective feelings become mass
quantified in a constant stream?
I have to address this. It makes all personal expressions
turn into categorized, quantified null data. Does this fit with
what I am trying to convey?
The value of social media as a device for expression
and its possibility to measure wellbeing.

Also this code does not ask for username/ password and
other Security details. I have run it for 30 mins without
problems. I want to know why it doesn’t require security
authorization and is this a risk?

I can add more search terms to one label. So happy could

search joy, laughter, pleasure, etc.

Atleast it counts all the tweets of people using that word.

5000 for ‘happy’ takes less than 5 minutes.

Need to turn it into an applet to embed online. Need also

to create a sketch that prints tweets.

This Bar Graph will be core to my economy system. Each

trader for each emotion will have their own feed for that
word. I would like to make a sketch that pulls out just the
text from tweets with that word every 3 seconds.

This information is useful, With this I now need to expand

and make an Economy. A system to buy and sell.

Basic economics and The Stock

Market
This graph is data that needs to be combined with the
audience buying and selling. So a graph that responds to
the frequency info and its selling rate. The frequency of
the emotion effects the price. This is what shall be shown
online. Each vendor can register a sale of an emotional
category. (need to make a manual program for creating a
collective graph that is broadcast online)

Another basic code for streaming tweets is called

‘simplestream’. Each vendors computer will stream this
with their search term.
Just entered this code to make it stream ‘sad’
"1/statuses/filter.json?track=sad"

now trying to work out if im going to be extending the API

limits by running 4 stream.twitter.com and (twitter.search)

The search API does not require authentiction. (bar chart

fine)
‘The Streaming API supports both basic and OAuth authentication
on  stream.twitter.com. For the time being there is no date on which basic
authentication will be turned off for the Streaming API so you are
free to choose whichever method you wish.’

API methods which are not directly rate limited are still subject to
organic, unpublished limits. This includes actions like publishing
status updates, direct messages, follow/unfollow actions, etc.
These Twitter Limits are described on our help site.

e.g. no api limit on publishing status

updates?

So streaming API is fine.

statuses/sample

Returns a random sample of all public statuses. The default

access level, ‘Spritzer’ provides a small proportion of the Firehose,
very roughly, 1% of all public statuses. The “Gardenhose” access
level provides a proportion more suitable for data mining and
research applications that desire a larger proportion to be
statistically significant sample. Currently Gardenhose returns, very
roughly, 10% of all public statuses. Note that these proportions are
subject to unannounced adjustment as traffic volume varies.
 URL: http://stream.twitter.com/1/statuses/sample.json
 Method(s): GET
 Parameters: count, delimited
 Returns: stream of status element
track

Specifies keywords to track. Phrases of keywords are specified by

a comma-separated list. Queries are subject to Track Limitations,
described in Track Limiting and subject to access roles, described
in the statuses/filter method.
Comma separated keywords and phrases are logical ORs,
phrases are logical ANDs. Words within phrases are delimited by
spaces. A tweet matches if any phrase matches. A phrase
matches if all of the words are present in the tweet. (e.g. ‘the
twitter’ is the AND twitter, and ‘the,twitter’ is the OR twitter.). Terms
are exact-matched, and also exact-matched ignoring punctuation.
Each comma-seperated term may be up to 60 characters long.
Exact matching on phrases, that is, keywords with spaces, is not
supported. Keywords containing punctuation will only exact match
tokens and, other than keywords prefixed by # and @, will tend to
never match. Non-space separated languages, such as CJK and
Arabic, are currently unsupported as tokenization occurs on
whitespace. Other UTF-8 phrases should exact match correctly,
but will not substitute similar characters to their least-common-
denominator. For all these cases, consider falling back to the
Search REST API.
Track examples:
 Twitter will return statuses which contain: TWITTER, twitter,

“Twitter”, twitter., #twitter and @twitter.

 Twitter will not return statuses which contain: TwitterTracker

and http://www.twitter.com.
 helm's-alee will return statuses which contain: helm’s-alee

 helm's-alee will not return statuses which contain: #helm’s-alee

 twitter api,twitter streaming will return statuses such as: “The Twitter

API is awesome” and “The twitter streaming deal is fast”.

 twitter api,twitter streaming will not return statuses such as: “I’m new

to Twitter”.
 chirp search,chirp streaming will return statuses such as: “Listening to

the @chirp talk on search”, “I’m at Chirp talking about search!”,

and “loving this search talk #chirp”.
Values: Strings separated by commas. Each string must be
between 1 and 60 bytes, inclusive.
Methods: statuses/filter
Example: 
Create a file called ‘tracking’ that contains, excluding the quotation
marks, the phrase: “track=basketball,football,baseball,footy,soccer” then execute:
curl -d @tracking http://stream.twitter.com/1/statuses/filter.json -uAnyTwitterUser:Password.

You will receive JSON updates about various crucial sportsball

topics and events.

2 things I need to sort out visually in code -

Search graph – add extra bar.
Stream – bar to restart when hits other side. (give co-
ordinate) If rect x (etc)

Access and Rate Limiting

All accounts may access
the statuses/sample and statuses/filter methods at default
access levels. Accounts may also be granted broader data access
on these same methods on a case-by-case basis. Access to other
methods requires a special arrangement with Twitter.
ContactContact Twitter with your use case, a brief description of
your organization, and the requested access level(s).
Each account may create only one standing connection to the
Streaming API. Subsequent connections from the same account
may cause previously established connections to be disconnected.
Excessive connection attempts, regardless of success, will
result in an automatic ban of the client's IP address.
Continually failing connections will result in your IP address
being blacklisted from all Twitter access.

I need an account for each streaming filter sketch.

http://dev.twitter.com/pages/streaming_api_concepts

still might be too crowded for one api to run

1 search enquiry (every 3 seconds)
3 samping statuses/filter 
28/04/2011
TWITTER SAYS ITS GOING TO BE OK

Still need to print in stream sketch and have bar graph

restart.
Design the share index graph a bit more.
Could use a status request query sketch to print tweets
but need another twitter account for that one station.

Max Dovey, May-19 06:29 am (PDT):

Hello, for an art installation i will be having 4 cpus running the search
twitter query (search.twitter.com) . because the rate limit is not made
public, if you could authorize my proposed request frequency. if i have each
computer (on the same IP adress) calling the search api every 10 seconds,
that comes to 1440 calls per hour (from the same Ip adress) whilst also
running the sample stream (on each cpu) . would this be allowed?
i know you dont whitelist search api users but it would be great to know
what procedures can keep me within the rules.
it would be between June 16th and June 24th, continuously during day time
hours.
thanks alot.
-- 
Max Dovey

sutorius, May-19 10:11 am (PDT):

Hello,
While the estimated rate of 1440 queries per hour from your single IP should
be within our limits, please consider using the Streaming API for all of your
needs and not just the sample:http://dev.twitter.com/pages/streaming_api .
For example, you can connect to a real-time stream of all tweets containing a
keyword using the statuses/filter method and the track parameter. We hope
that you are able to implement your app with the Streaming API, and are sorry
for any inconvenience that this may cause you and your team.
Thanks,
Twitter API Policy

Conclusion
The streaming Twitter part of the emotional stock market
was always going to be hard. Luckily I located a lot of
guides online and teachings from people better at
computer programming than me. I have customized a bar
graph for counting tweets and have also got a stream bar
that shows every tweet using that word. Fortunately it was
important to remember this was a piece of art, not social
science and could therefore be ambiguous. Its more
important to give the impression that every tweet is being
counted, than doing it.

Bibliography – (a list of helpful people and websites)

Daniel Solitis , Author of the Twitter Party Race
http://www.danielsoltis.com/Recent_Work.html

Jay Thor – Canadian creative data artist good guides

http://blog.blprnt.com/source-code-tutorial

Mark Mcbride – made the simple twitter stream for

processing.
http://mccv.github.com/processing-tweet-stream/

Processing forum –
http://forum.processing.org/

Tinker London – A guide to processing and Twitter

http://tinkerlondon.com/now/2010/09/14/oauth-twitter-and-
processing/

Twitter Developers – guides for app developers

http://dev.twitter.com/

Twitter 4j – a java application for twitter and processing

http://twitter4j.org/

RobotGrrl – creative hacker

http://robotgrrl.com/blog/2011/02/21/simple-processing-
twitter/