USOO946.

0695B2

(12) United States Patent (10) Patent No.: US 9,460,695 B2

Szalay et al. (45) Date of Patent: Oct. 4, 2016
(54) SYNTHESIZER WITH BI-DIRECTIONAL (56) References Cited
TRANSMISSION
U.S. PATENT DOCUMENTS
(71) Applicant: Fishman Transducers, Inc., Andover,
MA (US) 5,576,507 A 11/1996 LaMarra
5,834,671 A 11/1998 Phoenix
(72) Inventors: Andras Szalay, Budaors (HU); 7,304.232 B1* 12/2007 Nicholes .................. G1OH 1,46
Lawrence Fishman, Winchester, MA 84,741
(US) 7.427,710 B2 9, 2008 Hara
2002/0005108 A1* 1/2002 Ludwig .................... G1OH 1.00
(73) Assignee: FISHMAN TRANSDUCERS, INC., 84f600
Andover, MA (US) 2002/0124715 A1* 9/2002 Fay ................................. 84,645
2002/0178012 A1* 11/2002 Wang et al. ... TO4,503
(*) Notice: Subject to any disclaimer, the term of this 2003/0172797 A1* 9, 2003 JuSzkiewicz ......... G1OH 1,0058
patent is extended or adjusted under 35 84f601
U.S.C. 154(b) by 19 days. 2004/0187673 A1* 9, 2004 Stevenson ................ G1OH 1.02
84/737
(21) Appl. No.: 14/159,961 2006/O124735 A1 6/2006 Mizuno et al. ............... 235,382
2006/0196348 A1 9/2006 Cummings
(22) Filed: Jan. 21, 2014
(65) Prior Publication Data (Continued)
US 2014/020231.6 A1 Jul. 24, 2014 OTHER PUBLICATIONS
Related U.S. Application Data VJ Franz K. “Mid Air Wireless MIDI MAudio-Unboxing and
(60) Provisional application No. 61/754.293, filed on Jan. Demonstration,” https://www.youtube.com/
18, 2013. watch?v=DMoGWhha 2s. Mar. 2, 2010.
(51) Int. Cl. (Continued)
GIOH 3/00 (2006.01)
GIOH IMO (2006.01) Primary Examiner — Marlon Fletcher
GIOH 3/8 (2006.01) (74) Attorney, Agent, or Firm — Pearl Cohen Zedek
(52) U.S. Cl. Latzer Baratz LLP
CPC ............ G10H 1/0066 (2013.01); G10H 3/188
(2013.01); G 10H 2.220/106 (2013.01); G 10H (57) ABSTRACT
2240/211 (2013.01) An audio or visual system may include an encoder to encode
(58) Field of Classification Search electrical signals generated by an instrument such as a
CPC ............. G10H 1/0066; G10H 1/0058; G10H stringed instrument to music control message data such as
2240/056; G10H 1/0008: G10H 2240/311; MIDI data. A first wireless transceiver coupled to the
G10H 2240/211; G10H 3/18: G10H 1/0083: encoder may transmit the MIDI data to a second wireless
G10H 1/342; G10H 3/186; G10H 2240/031; transceiver. A processor, coupled to the second wireless
G10H 1/368; G10H 2210/031: G10H 1/186: transceiver, may produce media signals based on the MIDI
G10G 1/00; G09G 2370/16; G09G 3/002: data.
GO8C 23/04
See application file for complete search history. 20 Claims, 9 Drawing Sheets

5 10a | 1 a h
11) 11c
-Illd 4
le
f

14a
 US 9,460,695 B2
Page 2

(56) References Cited 2014/0053712 A1 2/2014 Dreher ............................ 84,645

2014/O123838 A1* 5, 2014 D'Amours ...................... 84f626
U.S. PATENT DOCUMENTS 2014/0202316 A1* 7/2014 Szalay ................. G1OH 1/0066
84,645
2007/OOOO375 A1 1/2007 Harrison
2007,0003073 A1 1/2007 Iriarte
2007/0227335 A1 * 10, 2007 Fukada et al. .................. 84f600 OTHER PUBLICATIONS
2007/0234880 A1 10, 2007 Adams et al. && 7 s
201O/OO37755 A1 ck 2, 2010 McMillen . . . . . . . . . . . . . G1OH 1,125 Tetrastyle, Wireless MIDI Transmitting by XBee (experiment)
84,645 https://www.youtube.com/watch?v=DMoGWhha 2s. May 26,
2010, O234109 A1 9, 2010 Chiu et al. 2011.
2011 0003638 A1 1/2011 Lee et al. Pandamidi Solutions “midiBeam' http://web.archive.org/web/
2011/0028218 A1 2/2011 Gomes et al. .................. 463,39 20131211070233/http://pandamidi.com.midibeam? Dec. 11, 2013.
2011/0174138 A1* 7, 2011 Loh et al. ....................... 84,645
2013/0087037 A1* 4/2013 Dreher ............................ 84,645 * cited by examiner
U.S. Patent Oct. 4, 2016 Sheet 1 of 9 US 9,460,695 B2

I"OIH
U.S. Patent Oct. 4, 2016 Sheet 2 of 9 US 9,460,695 B2

I
C
>
-

H
s
C.
•
2

S
U.S. Patent Oct. 4, 2016 Sheet 3 of 9 US 9,460,695 B2

K9BI0dsI5G

Z09)

[07
U.S. Patent Oct. 4, 2016 Sheet 4 of 9 US 9,460,695 B2

-----------
U.S. Patent Oct. 4, 2016 Sheet 5 Of 9 US 9,460,695 B2

----------------------------------------------------------------------------
U.S. Patent Oct. 4, 2016 Sheet 6 of 9 US 9,460,695 B2

r e
"--a
U.S. Patent Oct. 4 9 2016 Sheet 8 of 9 US 9,460,695 B2

-.
i. autumnistentinuatuantinue rara
--

& C 33 3:33: 2.

?§
-!p) 8-09

009
 US 9,460,695 B2
1. 2
SYNTHESIZER WITH B-DIRECTIONAL FIG. 5 is an example user interface for editing MIDI
TRANSMISSION parameters, according to embodiments of the invention.
FIG. 6 is a user interface for editing MIDI parameters and
PRIORAPPLICATION DATA mixing audio signals, according to embodiments of the
invention.
This application claims the benefit of prior U.S. Provi FIG. 7 is a flowchart of a method according to embodi
sional Application Ser. No. 61/754.293, filed Jan. 18, 2013, ments of the invention.
which is incorporated by reference herein in its entirety. It will be appreciated that for simplicity and clarity of
illustration, elements shown in the figures have not neces
FIELD OF THE PRESENT INVENTION 10 sarily been drawn to Scale. For example, the dimensions of
Some of the elements may be exaggerated relative to other
The present invention relates to guitar synthesizers or elements for clarity. Further, where considered appropriate,
other synthesizers that may be played with other instru reference numerals may be repeated among the figures to
mentS. indicate corresponding or analogous elements.
15
BACKGROUND DETAILED DESCRIPTION

Keyboard synthesizers may be well-known tools for cre In the following description, various aspects of the pres
ent invention will be described. For purposes of explanation,
ating music control message data such as MIDI data or notes specific configurations and details are set forth in order to
that may be converted to synthesized or sampled sounds. For provide a thorough understanding of the present invention.
guitar synthesizers or other instruments, the setup may be However, it will also be apparent to one skilled in the art that
more complicated. For example, on a guitar, a separate MIDI the present invention may be practiced without the specific
converter box may be coupled directly to the guitar through details presented herein. Furthermore, well known features
a cord. The connection between the guitar and the external 25 may be omitted or simplified in order not to obscure the
box can be a multiplexed analog signal (as used by the present invention.
Shadow GTM-6 and Passac Sentient Six MIDI controller Unless specifically stated otherwise, as apparent from the
boxes) or a unique multi-wire cable (such as WL Pitchrider, following discussions, it is appreciated that throughout the
Korg Z3, and K-Muse Photon MIDI controllers), a standard specification discussions utilizing terms such as “process
24 pin multi-wire cable (such as Roland or Ibanez IMG 30 ing.” “computing. "calculating,” “determining,” or the like,
2010 MIDI controller boxes), or a 13 pin cable (such as refer to the action and/or processes of a computer or com
Yamaha G50 or Axon MIDI controller boxes). However, puting system, or similar electronic computing device, that
during performance, musicians may be tethered to these manipulates and/or transforms data represented as physical,
kinds of boxes. A way to allow a musicians freedom during Such as electronic, quantities within the computing systems
performance and maintain low latency in converting sounds 35 registers and/or memories into other data similarly repre
to MIDI may be needed. sented as physical quantities within the computing systems
memories, registers or other Such information storage, trans
SUMMARY mission or display devices.
Embodiments of the invention may provide a system or
An audio or visual system may include an encoder to 40 method for producing media signals based on an instrumen
encode electrical signals generated by an instrument to talist’s actions on an instrument, including an acoustic,
music control message data Such as MIDI data. A first electrical, or electronic musical instrument. Such as an
wireless transceiver coupled to the encoder may transmit the electric guitar, acoustic guitar, electric bass, acoustic violin,
MIDI data to a second wireless transceiver. A processor, flute, or clarinet, for example. The media signals may be
coupled to the second wireless transceiver, may produce 45 audio or video that may be samples from existing record
media signals (e.g. audio signals, video) based on the MIDI ings, audio signals synthesized using synthesizing hardware
data. or software, signals that direct a configuration of lighting
effects on stage, or other signals that may control or direct
BRIEF DESCRIPTION OF THE DRAWINGS an audiovisual performance or display. Actions on an instru
50 ment may be converted to data that conform to a format Such
The Subject matter regarded as the invention is particu as a standard Music Instrument Digital Interface (MIDI)
larly pointed out and distinctly claimed in the concluding format, an electronic musical instrument industry data for
portion of the specification. The invention, however, both as mat specification that enables a wide variety of digital
to organization and method of operation, together with musical instruments, computers, synthesizers, and other
objects, features, and advantages thereof, may best be under 55 related devices to connect and communicate with one
stood by reference to the following detailed description another. The data or MIDI data may include information
when read with the accompanying drawings in which: about pitch, Volume, and a length of time that a sound is
FIG. 1 is a diagram of an audio or visual system, accord Sustained, for example. The musical usage of a guitar
ing to embodiments of the invention. synthesizer system may require a complex structure of
FIG. 2 is a schematic diagram of an audio or visual system 60 parameters that determine how the Sound responds to the
using a standalone receiver box, according to embodiments actions of the guitarist. Such a set of parameters may
of the invention. describe splits between different sounds according to the fret
FIG. 3 is a schematic diagram of an audio or visual system range or the string range that is played, the response to
using a personal computer, according to embodiments of the picking strength, or the limit of picking which triggers a
invention. 65 MIDI note at all, and many other parameters. Such a set is
FIGS. 4A-4D are illustrations of an encoder and pickup, called in MIDI terminology for example “preset', or
according to embodiments of the invention. “patch', or “program'. Musicians may use different patches
 US 9,460,695 B2
3 4
typically for each song, but often several patches may be equipped with an encoder to encode or convert the electrical
required even within one song. Within each patch, there may signals output by the pickup into MIDI data. The encoder
be multiple splits, which divide sound characteristics may contain an analog to digital (A/D) converter (ADC) that
depending on which notes are played. For example, in one converts the analog electrical signal to a digital format that
patch, a lower octave played may be characterized by piano is then that can then be processed by a digital signal
Sounds, and a high octave played may be characterized by processing (DSP) device, processor, or microprocessor, for
violin Sounds. Other configurations may be used. A set of example. Alternatively, the ADC may be coupled to the
parameters or patch may be data stored in a memory. pickup. The analog processing on the encoder may process
The produced media signals may be media samples or the electrical signals using a pitch detection algorithm that
synthesized sounds that are controlled by the music control 10 calculates the musical pitch produced by the musical instru
data (e.g. MIDI data), for example, and may be produced ment. This pitch information may be converted to a Midi
having different sound qualities from the instrument that the Note Number, or other control message, that is wirelessly
instrumentalist is playing on. For example, the instrumen transmitted to the receiving device. This Midi Note Number
talist may be playing on a guitar, and the actions on the may determine, for example, the pitch of the note that may
guitar may be converted to MIDI data, and the MIDI data 15 be played by the Sound producing device on the output
may be wirelessly transmitted and used to trigger or control module.
a sampled or synthesized piano Sound or a synthesized flute For Stringed instruments, each string may have vibrations
sound on another device. Other types of sound may be detected individually and may provide a data channel (e.g.
triggered or controlled, which may emulate other instru a MIDI data channel or other music data channel) that can
ments, noise, speaking, or electronically generated Sounds, be processed independently from that of other strings. In
for example. Video recordings or samples may also be particular, an electric guitar having six strings may provide
triggered by the music control data, control signal, control six MIDI data channels. A pickup may sense or detect the
message, or MIDI data. For example, a guitarist's actions on vibrations on each of the six strings. The encoder may
the guitar may trigger certain video images to be displayed include six separate ADCs to convert each strings vibra
in desired parts of a song, for example. The music control 25 tional information to a digital format, which may then be
data (e.g. control signal, control message or MIDI data) may multiplexed or combined to be processed by the DSP. The
control lighting effects on a stage. Such as laser light effects, fret or note positions on each string may be further divided
strobe light effects, color effects, or other lighting effects that into splits having different sound characteristics, for
may be seen during a performance. Data formats for com example. Unlike a piano keyboard, where each note can be
municating with devices including music or note informa 30 programmed with MIDI information or messages, the same
tion or control messages (e.g., event messages specifying guitar note can be played on different strings (e.g., an A note
notation, pitch and velocity, control signals for parameters at 220 Hz may be played on the second fret of the G string
Such as Volume, vibrato, audio panning, cues, and clock or the seventh fret on the D string), and it may not be
signals) other than MIDI may be used. practical for specific notes to be assigned different MIDI
A synthesizer, e.g., a MIDI synthesizer, for example, may 35 settings. Separately converting each string into MIDI data
receive music data or note information such as MIDI data may provide a guitar player with a wide range of playability.
and output audio signals. Other musical or notation stan In addition to Midi Note Number, other control messages
dards, or data formats for transmitting music or control may be generated by the DSP that define the dynamic
messages, may be used. Though some embodiments behavior of the note that is produced by the output module.
described herein are directed primarily to a guitar, the 40 This dynamic control information describes the musical
claimed invention may be further applicable to other acous nuances of the notes as they are played on the instrument.
tic or electric musical instruments, whose Sound may be Examples of these control messages are: Pitch Bend or
converted to electrical signals through a guitar or other Velocity or specifying a particular instrument Voice that
stringed instrument pickup, for example. Further, embodi should be played.
ments of the invention may allow wireless transmission of 45 Parameters may further be defined that determine the way
data between a musical instrument and a receiver which may that the MIDI encoder or other music information encoder
be connected to a speaker or amplifier. Wireless transmission responds to the actions of an instrumentalist or player of an
may occur over any wireless custom non-standard protocol, instrument (e.g. a guitarist). These parameters may set
such as the consumer bandwidth of 2.4 GHz, or over a boundaries that are used by the (DSP or other processor
standard protocol, such as IEEE 802.11, Bluetooth, or Wi-Fi, 50 coupled to the encoder) in determining the correct values
for example, and may communicate over different radio that are output as control messages. Some examples of these
bands, such as the industrial, scientific and medical (ISM) boundaries may include: Note On value what minimum
radio bands. excitation of the musical instrument that may represent a
Embodiments of the invention may allow processing of legitimate note on event, Note Off value what minimum
analog audio signals for the output of MIDI data. The 55 vibrational level that may determine a legitimate note off
processing may occur on the musical instrument itself and event, Pitch Bend range how pitch modification may be
the MIDI data output may be transmitted wirelessly to a produced by the Sound producing module in response to the
speaker, amplifier, analyzer, or other equipment and output actual pitch bend produced on the musical instrument,
devices that may be able to further read and process MIDI Volume control messages—messages that may follow the
data. The musical instrument may be equipped with a 60 envelope of the note produced by the musical instrument
pickup. The pickup may be, for example, a magnet coil that are sent to the output device to control the volume of the
pickup, a piezoelectric pickup, a microphone, an acceler Sound produced, Quantization—settings that determine how
ometer, an optical pickup, or any other device that translates to convert detected pitches that fall between conventional
vibrational information generated by the musical instrument notes, or Dynamic Sensitivity, which may control how the
into an electrical signal that is representative of the vibra 65 encoder interprets Volume variations in a musician’s play
tions when measured as the magnitude of the signal with ing. The values of these parameters may be set according to
respect to time. The musical instrument may also be the way the user plays an instrument or the way a user wants
 US 9,460,695 B2
5 6
their playing to Sound. These parameters may be global in direction may be primarily used, from the guitar towards the
nature such as general input sensitivity, tuning base (e.g., receiver box/sound generator, but a backwards communica
whether an A is at 440 Hz. (A440) or 441 Hz. (A441), etc.). tion may provide further benefits. Although it would be
These may also be specifically set to complement a particu possible to construct a system that consists of a relatively
lar Sound that is being played, such as turning off pitch bend “dumb' transmitter on the guitar, raw data may need to be
when playing a piano Sound. The set of parameters that are modified according to the actual patch on the receiver side.
not global may be assigned to a “preset,” “patch,” or other This may have the consequence that the “intelligence' of the
program that bundles these control messages with a particu system is divided between the guitar device and the receiver.
lar sound that is assigned to a particular MIDI channel. This may have several disadvantages: higher software devel
These parameters or patches may be stored in memory in the 10 opment effort for each receiver option separately; higher
encoder. The encoder may provide knobs and buttons or cost for the receivers with stronger processors and larger
other controls to adjust the patches. Alternatively, the memory; compromises that cannot be resolved, since some
encoder may be in communication with a user interface patch parameters (e.g. pick trigger sensitivity) must influ
separate from the encoder which allows a user to change ence signal processing that may take place in the guitar.
parameters on the user interface. The patches may alterna 15 Instead, it may be more practical to concentrate the intelli
tively be stored in the memory of the output module and gence of the system in a central location, Such as on the
communicate wirelessly back to the encoder when a param guitar unit. Thus, all kind of modifiers (foot Switches, pedals,
eter value is changed. remote control) located on a receiver box may have a
Embodiments of the invention may allow editing or backwards data path into the central unit on a guitar. Patches
manipulation of the signals being transmitted from the may also be stored in the central unit, with a way to archive
guitar. The guitar may include an encoder which encodes them on a computer, and it may be possible to reload them
signals from the guitar into MIDI data. The MIDI data may from the computer to the guitar using the backwards data
be sent wirelessly, e.g. via radio, to a computer or other path. Embodiments of the invention may encompass wire
device with editing or synthesizer software on it. The guitar less unidirectional transmission of data (e.g., from a trans
itself may have knobs, buttons, and potentiometers that may 25 mitter on a guitar to a receiver coupled to a receiver box or
manipulate sounds or audio signals produced by the guitar. computer) or wireless bi-directional transmission of data
One or more user interfaces may be provided which may be (e.g., two way communication between a transmitter on a
accessible through a computer. The user interface may guitar and a receiver).
indicate or visualize parameters that are being manipulated Most data transmission chipsets may include a way of
by the guitar or the computer itself. A transmitter and 30 handshaking between the transmitter and the receiver: the
receiver may have the capability to communicate bi-direc receiver may send back an acknowledge signal to the
tionally (each sending data to and receiving data from the transmitter, so the transmitter can be sure that the message
other), as transceivers. The parameters stored in the encoder has arrived and does not have to be repeated. In the chipset
may be changed by controls on the encoder or by controls on used in Some embodiments, there may be the additional
the user interface couple to the receiver. When new param 35 possibility to hide a user message in the acknowledge signal.
eters are to be stored in the encoder, the receiver may Thus, it is possible to send data backwards from the receiver
wirelessly transmit the new parameters to the encoder. The to the transmitter, but communication may not be purely
encoder may then save the new parameters. The parameters symmetrical: initiation may be only performed by the trans
may be further stored in a memory coupled to receiver, Such mitter, and the receiver can pack its data in the answer to the
as the memory of a computing device. These parameters 40 initiation.
may be changed by the user interface or by controls on the In the guitar synthesizer System the latency of the sounds
encoder. When new parameters are to be stored in the user may be a critical parameter, and may generally be kept to a
interface, the encoder may wirelessly transmit the new minimum. If the latency of the backwards communication is
parameters to the computer. The computer may then save the also kept within reasonable limits (which does not have to be
new parameters. The parameters may be stored in the 45 as Small as for the transmitter-to-receiver communication)
encoder and the computer simultaneously. The receiver then the system may be just as usable as if it would have
(e.g., the receiver coupled to a computing device) may wired bi-directional connection. The reasonable latency for
communicate with the transmitter through a protocol that the backwards communication may be limited by real-time
reduces error in transmission. The protocol may allow full actions like pressing a foot Switch, for example. If back
syncing of parameters between the encoder and the user 50 wards communication (e.g., from the receiver box to the
interface. guitar) gets through with a latency of not more than about 10
A transmitter may be located on the musical instrument, milliseconds, then the sensation of latency may not appear
and coupled to an encoder which converts electrical signals for the guitarist; it may appear as real-time. Therefore,
received from the pickup to MIDI data. The receiver may embodiments may be constructed in a way that if the
send an acknowledgement signal to the transmitter so that 55 transmitter has no data to send in a time of 7 milliseconds,
the transmitter can confirm that a connection exists between then it may send out a dummy message, in order to provide
the receiver and the transmitter. The transmitter may be the a way for the receiver to send back its message. In this way,
device that always initiates communication with the the receiver may send a new data package to the transmitter
receiver. The hardware in the transmitter and receiver may in not more than 7 milliseconds. At the same time the
maintain low latency in creating and transmitting MIDI data, 60 message from the transmitter may serve the purpose of
so that the guitarist or instrumentalist can maintain a natural sending out a “I am alive” message to the receiver (“Active
feel of the instrument while performing or recording with Sensing in MIDI terminology) that may provide a way to
the embodiments herein. A user may also initiate pairing turn off hanging notes on the Sound generators if commu
between the transmitter and receiver. nication between the transmitter and receiver breaks down
Radio circuitry used may be capable of communicating in 65 for any reason. Other latencies may be used.
one direction at a time only, either as a transmitter or as a FIG. 1 is a diagram of an audio or visual system, accord
receiver. In case of the wireless guitar synthesizer only one ing to embodiments of the invention. On an instrument 102
 US 9,460,695 B2
7 8
Such as an electric guitar, a pickup 100 may detect vibrations bi-directional data transmission may be possible between
from strings on the instrument 102 for example, due to the guitar 102 and computer 110. For example, a user on
pickup’s 100 close proximity to the instrument's 102 strings computer 110 may choose or decide that a C note played on
103. The pickup 100 may convert these vibrations to elec a low E String should sound like an electric violin Sound
trical signals and send electrical signals to an encoder 104. 5 played at a high Volume and Sustained. The user may input
The electrical signals may first be analog and processed the MIDI parameters 115 via an input device 118. The MIDI
through an analog/digital (A/D) converter to convert them to parameters 115 may be transmitted to transceiver 106 on
digital signals for further processing. The vibration of each guitar 102 and stored in the encoder's memory 104a. When
string 103 may be processed through a separate ADC and the user plays the C note on the particular string (but not
then sent to a DSP. The encoder 104, including a memory 10 necessarily on another string of the same guitar), the pickup
104a and processor 104b, may encode or convert the elec may detect the string's vibration and the encoder’s ADC
trical signals from the pickup 100 into MIDI data or other may convert the electrical signal to digital signal. The
kinds of musical note or music control messages or data. The encoder's DSP may convert the digital signal and generate
encoder may include an A/D converter, or alternatively, the or create a MIDI message or control message indicating a C
A/D converter may be located on the pickup 100. The 15 note that should be played like an electric violin with a high
encoder 104 may be coupled to a transmitter or transceiver Volume value and Sustained. During play, the MIDI message
106. The transceiver 106 may transmit the MIDI data or may be transmitted from the transmitter-transceiver 106 to
music control messages or data wirelessly (e.g. via radio) to receiver 108. The synthesizer 111d, via processor 110b, may
a receiver or a second transceiver 108. The receiver-trans receive the MIDI message and generate an audio signal
ceiver 108 may be a Universal Serial Bus (USB) device according to the MIDI message’s instruction to an output
connectable to a computer 110, for example. Alternatively, device 114 (Such as a speaker or amplifier) that Sounds
the receiver-transceiver 108 may be embedded in a stomp similar to an electric violin playing a C note loudly and for
box or standalone receiver box. a longer time than is typical for the Sound produced via one
The computer 110 may include memory 110a and a guitar pluck. Additionally, sampler 111e may produce video
processor 110b. Memory 110a may store software such as a 25 signals from stored video samples or other stored images
digital workstation 111a, audio editor 111b, and audio mixer (e.g., computer graphics) to output device 114. Output
111c, for example. Memory 110a may also include software device 114 may include a display 114a to play video clips or
for synthesizers 111.d or samplers 111e. Memory 110a may signals based on music control data (e.g., control signals,
further include software for editing or visualizing MIDI control messages or MIDI messages) received by receiver
parameters. Such programs may include or be compatible 30 108.
with Avid's Pro Tools, Apple's GarageBand and Logic Processor 110b may execute software or code to carry out
software, Steinberg's Cubase software, Ableton Live soft embodiments of the invention. For example, processor 110b
ware, and Presonus's Studio One software. The computer may act as synthesizers 111d. Samplers 111e, workstation
110 may include a display 116 that allows or enables a user 111a, audio editor 111b, or audio mixer 111C. Computer 110
to edit MIDI parameters for encoding electrical signals from 35 may be a typical consumer PC or other laptop with software
the pickup 100 to MIDI data. Processors 110b and 104b may loaded to it, or computer 110 may be a standalone computing
each carry out all or part of embodiments of a method as device or receiver box that implements real-time audio
discussed herein, or may be configured to carry out embodi mixing and editing tasks and may be particularly Suited for
ments, for example, being associated with or connected to a use during musical performances, for example.
memory 110a and 104a storing code or software which, 40 FIG. 2 is a schematic diagram of an audio or visual system
when executed by the processor, cause the processor to carry using a standalone receiver box, according to embodiments
out embodiments of the method. of the invention. Pickup 200 may send data from a guitar 201
The synthesizer 111.d or sampler 111e may be separate or to an encoder 202 which is also mounted on a guitar. Pickup
integrated with computer 110. The synthesizer 111d may 200 and encoder 202 may be removably attached to the
generate, e.g. by processor 110b, media signals such as 45 guitar 201 during performance. Pickup 200 and encoder 202
audio signals based on the received MIDI data or musical may include adhesive material, such as glue or VelcroTM, or
note or music control data or messages from the receiver 108 be magnetic, and be able stick onto the guitar while a
and the parameters selected on digital workstation 111a. musician is playing. Pickup 200 and encoder 202 may be
Such as which type of instrument Sound to generate (e.g., able to be removed if a musician does not wish to use the
electric violin). The sampler 111e may store a set of recorded 50 synthesizer system. Encoder 202 may alternatively be con
Sounds or video clips or other instructions (e.g. lighting nectable to a standard pickup 200 that both may be originally
control instructions) in memory and produce audio or video manufactured with, embedded in, or integral to the guitar
signals that replay the recorded sounds or video. The data 201.
received from receiver 108 may dictate which recorded The encoder 202 may include an ADC 203 to convert
sound to play. The digital workstation 111a may further 55 analog electrical signals from the pickup 200 to digital data
control the way that the recorded sounds are played (e.g., or signals. Encoder 202 may further include a processor 204
with a high pass filter). for processing the digital data from the ADC 203. The
The computer 110 (e.g. via a user interface shown on processor may convert or encode the digital data originating
display 116 or input devices such as a keyboard 118) may from the pickup 200 into MIDI data or other data. The
allow the setting of music control message data parameters 60 encoder 202 may include memory 205 to store MIDI param
(e.g. MIDI parameters) 115 such as for example volume or eters that affect how digital data from the ADC 203 is
reverb, for example. These parameters 115 may be saved or converted to MIDI data. MIDI parameters may include, for
stored in computer memory 110a. The computer 110 may example, volume, quantization, or pitch bends. The MIDI
further wirelessly transmit the music control message data or data may include information Such as the frequency of a
MIDI parameters 115 through receiver-transceiver 108 to 65 pitch and the length of time that a pitch is Sustained. The
transmitter-transceiver 106 on guitar 102. The parameters encoder 202 may be coupled to a wireless (e.g., radio)
115 may be stored onto the encoder's memory 104a. Thus, transceiver 206. Control elements 208 may be included in
 US 9,460,695 B2
9 10
the encoder 202 to select MIDI parameters or sets of MIDI one by one, and then repeating from the beginning. After
parameters (e.g., patches) that affect the processing of audio sending the “I am here' message, transceiver 206 may
data to MIDI data. The control elements 208 may include evaluate if a second transceiver (e.g., 210 or 300) has hidden
push buttons and potentiometers, for example. an “I hear you' message in the acknowledge signal that
The transceiver 206 may transmit or send MIDI data to a answers transceiver 206's message. If the acknowledgment
second (e.g., radio) transceiver 210. The receiver may be signal is recognized, the pairing process may be completed
integrated in a Stomp box or standalone receiver box 212. with a "pairing finished' message transmitted to the receiv
The receiver box 212 may be a standalone device with a ers 210 or 300, and transceiver 206 may return to normal
processor 214a and memory 214b. The receiver box 212 transmission mode. After receiving the “pairing finished
may include Switches and pedals or other control elements 10 message, the receiver 210 or 300 may switchback to normal
216 to control functions such as hold, arpeggio, looper, or receive mode, and data communication may begin. The
other patches or sets of MIDI parameters. The receiver box channel settings of both devices may be automatically stored
212 may be configured or optimized for easy use during after a pairing, and will be recalled on next power up.
performance. The receiver box 212 may be connected to a FIGS. 4A and 4B are illustrations of a guitar pickup 400,
synthesizer 218 to generate sounds based on the received 15 according to embodiments of the invention. A guitar pickup
MIDI data and patches enabled by the switches on the stomp 400 may sense or detect vibrations from a guitar string as it
box. 212. The stomp box may include a display 215 that is plucked. The pickup 400 may be mounted directly on the
includes or generates a user interface 215a to display guitar, either by a user or embedded within the guitar at a
information and allow a user to edit or manipulate the MIDI time of manufacture. The pickup 400 may include a sensing
data received by the receiver. The user interface 215a coil unit 402 for each string that is being detected, for
including a touch pad or other inputs may further allow a example. Each sensing coil unit 402 may include a wire coil
user to edit MIDI parameters for encoding electrical signals 404 or other kind of coil (e.g., a printed coil) wrapped
to MIDI data and to allow a user to transmit a set of MIDI around a magnetic bar 406, which may have a magnetic field
parameters to the encoder 202. Alternatively, controls 216 around it. As a metallic or soft metallic string vibrates near
may be integrated with user interface 215a and vice versa. 25 the magnetic bar 406, the vibrations may change the mag
Encoder 202 may store the received MIDI parameters from netic field around the magnetic bar 406 and induce a current
receiver box. 212 as separate sets or patches in memory 205. within the wire coil 404. The current within the wire coil 404
During performance, for example, a musician may quickly may be transmitted or sent to an encoder or processor via a
select different patches stored in memory 205 through wire or connection 407 with the wire coil 404.
manipulating controls 208. In another example, patches may 30 FIG. 4C is an illustration of an encoder 408 and pickup
be saved in memory 214b on receiver box 212, and a 400, according to embodiments of the invention. The pickup
musician may manipulate controls 216 on the receiver box 400 may sense the sounds or vibrations of a nearby string on
to access different patches saved in the encoder's 202 an instrument. The encoder 408 may include several controls
memory. In this way, embodiments of the invention may to adjust MIDI parameters or other parameters. A volume
allow syncing of MIDI parameters between the encoder 202 35 knob 410 may set volume levels for each virtual instrument.
and the receiver box. 212. A guitar/synth selector 412 Switch may control which chan
FIG. 3 is a schematic diagram of an audio or visual system nels or voices are heard when the final synthesized sounds
using a personal computer, according to embodiments of the are produced. In a middle position, for example, a guitar
invention. The guitar 201, pickup 200, and encoder 202 may Voice and additional synthesized voices may be heard
include similar or the same elements and have similar 40 together. With guitar mode selected, the “synth channels
configuration as described in FIGS. 1 and 2. In some may be muted, and only the guitar's sounds may be heard.
embodiments, transceiver 206 may transmit MIDI data or With synth mode selected, the guitar channel may be muted,
control data to a pen-drive or USB-drive acting as a receiver and only virtual instruments may be heard. A set of control
300. The pen-drive receiver may be connected to a computer buttons 420 may allow navigation of a user interface or
302. Such as a laptop computer or desktop computer. The 45 patch editing Software on a separate computing device. A
computer 302 may include a processor 303a and memory status light 422 may verify battery power and the connection
303b to implement software, such as a software synthesizer between encoder and a receiver 426. A charge indicator LED
304 or sampler. The software synthesizer 304 may work or status light 424 may indicate when the encoder needs to
with or be compatible with audio editing or audio mixing be recharged. A receiver LED or status light may indicate or
Software, which may also be implemented by processor 50 verify when the encoder is scanning for a connection with a
303a and memory 303b. A display 306 or user interface wireless receiver 426. Other controls may be present on the
306a may allow a user to input MIDI parameters that affect encoder. The wireless receiver 426 may be a USB key or
the conversion of electrical signals from pickup 200 to MIDI microUSB key that may be compatible with a computer or
data. The display 306 or user interface 306a may work with computer system (e.g., 212 or 302). The wireless receiver
input or control devices 308, such as computer keyboards or 55 426 may allow the encoder to transmit MIDI data to the
a mouse. Instead of being a USB pen drive, receiver 300 computer for synthesis, for example.
may be embedded or integrated on the computer, Such as an FIG. 4D illustrates a mounting device 440 on a guitar 439,
internal wireless card, for example. Audio signals generated according to embodiments of the invention. The mounting
by synthesizer 304 and processor 303a may be output to a device 440 may be fixedly or stiffly attached to the guitar
speaker or amplifier, or other output device 310. 60 439. An encoder may be attached to the guitar or instrument
Since data transmission of MIDI data may be wireless, through mounting device 440. Magnets 442 may be located
pairing may need to be performed between transmitter on the mounting device 440 to secure the encoder. The
transceiver 206 and receiver-transceiver 210 or 300 in order mounting device 440 may allow the user to removably
for communication to occur on the same channel or fre attach the entire instrument portion (e.g., encoder and
quency. Upon the initiation of pairing, the transmitter may 65 pickup) of the system to the instrument without damaging or
being to send "I am here' messages on all available channels altering the instrument. The mounting device 440 also
one by one, for a short time on each channel, incrementing allows the encoder and pickup to be removed from the
 US 9,460,695 B2
11 12
instrument when they are not being used. The pickup may your virtual instruments. Quantization mode settings
also include a separate mounting system for removably can't override these individual plug-in settings.) in one
attaching it to the guitar 439 or instrument, or adjusting its embodiment there are four possible settings:
closeness to the strings. In other embodiments, the pickup Off. Notes are not rounded to the nearest half-step.
and encoder may be embedded within a guitar or other On. Notes may be rounded to the nearest half-step.
instrument at the time of manufacture. Auto: A compromise between Quantize On and Quan
FIG. 5 is an example user interface 500 for editing MIDI tize Off modes. Small pitch discrepancies may be
parameters, according to embodiments of the invention. The ignored, similar to Quantize On mode. But if a pitch
user interface 500 may be integrated with a display on a change seems more deliberate by a user, as in a
computer or a standalone receiver box (see, e.g., FIGS. 2 and 10 note-bend, Quantize Off mode may be used, and the
3). The user interface 500 may allow users to edit MIDI or pitch of bends are reproduced.
control parameters, or edit patches, which may be a set of Trigger. In this mode, no bends may be used. If, for
MIDI or control parameters. The user may then transmit the example, if a user bends the note Cup to D-flat, this
patch to an encoder mounted on a guitar. Some control or may be interpreted as two separate notes with two
MIDI parameters may include, for example (other param 15 separate attacks. This may be the best choice when
eters may be used): mimicking instruments such as piano and organ,
Mode (e.g. MIDI Mode) Selector 502: Switchable which may be unable to produce pitches that fall
between mono and poly. In poly mode, all channels between adjacent half-steps.
(e.g., notes from all six strings of a guitar) may be sent FIG. 6 is a user interface 600 for editing music control
with the same MIDI control messages. For example, all message parameters such as MIDI parameters and for mix
six strings of a guitar may be subject to the same pitch ing audio signals, according to embodiments of the inven
bend messages. In mono mode, each channel (e.g., each tion. User interface 600 may be displayed on a computer
string) may include its own MIDI messages. For system (e.g., 110 or 302) or a receiver box (e.g., 212), for
example, pitch bend may only apply to one of the example. A patch readout area 602 may allow a user to
Strings. 25 preview, select, load, and save patches, for example to a
Touch Sensitivity Control 504: Sets the dynamic response computer or the encoder. A sensitivity adjustment area 604
independently for each patch. may allow users to adjust dynamic sensitivity for each string
Pick/Fingerstyle Selector 506: Optimizes the touch 605 on a guitar. A mixer area 606 may allow a user to adjust
response for pick or fingerstyle playing. the Volume levels, panning, and solo/mute status of the
Sustain Pedal 508: sustains any note played (e.g., length 30 guitar and synth sounds that may be included each patch. A
ens the time stamp of a note). fretboard/splits area 608 may display each note played in
Sound Badge 510: Displays the channel or voice of the real time and may allow a user to create “splits' patches
patch. that assign different sounds to different parts of the fret
Dynamics Sensitivity Slider 512. Controls how the board. For example, as shown, a patch 612 may be titled
encoder interprets Volume variations during playing. 35 “Cadaver Bass'. The patch 612 may include two voices,
MIDI instruments may interpret volume on a scale of 0 “guitar 614 and “synth1 616, which may be assigned to
to 127. With Dynamic Sensitivity on its rightmost two different areas 614a and 616a on the fretboard 608. For
setting, the maximum dynamic range may allow the each voice, different sensitivity levels 604 may be set for
loudest notes to transmit a value close to 127 (as loud each string 605. The volume levels may be adjusted between
as possible in one system), and softest notes may be 40 'guitar” and 'synthl', e.g., the guitar may be at a less
closer to 0 (silence). With the slider at its center setting, Volume than the synth. The Cadaver Bass patch settings may
every note may transmits a fixed value of for example be sent to an encoder on a guitar. As a musician plays the
64; no matter how heavily or softly the instrument is guitar, the notes that correspond to area 614a on the fret
played, all notes may have the same level. This may be board 608 may produce a guitar sound and the notes that
useful when mimicking instruments with tones that do 45 correspond to area 616a on fretboard 608 may produce a
not change according to how hard a user plays, such as synth sound. Other settings that are assigned to the areas
organs and harpsichords. may be sent as control data Such as MIDI control messages
Dynamics Offset Slider 514. Shifts the entire dynamic by the encoder to a receiver. The fretboard 608 may also
scale (as defined by the Dynamic Sensitivity Slider) by allow a user to assign audio or video samples or other audio
e.g. itó4. The relative dynamic values are unchanged— 50 or visual effects to particular areas of a guitar, so that when
everything gets louder or softer depending on the a user plays on the associated area on the guitar, it is possible
setting. (This would be useful if, say, you wanted a for audio or video samples to concurrently play with the
fixed-volume sound at a dynamic level other than the user. The user may also assign commands that control
default setting of 64.) lighting, e.g. stage lighting, effects.
Transpose Control 516. A user can transpose each synth 55 FIG. 7 is a flowchart of a method according to embodi
independently from the others. Adjustable e.g. by ments of the invention. In operation 702, a musical instru
t1-24, with 1 representing a half-step, and 24 repre ment may generate electrical signals. This may occur
senting the maximum transposition of two octaves. (A through a pickup attached to the instrument, and the pickup
user might, for example, dial in a setting of -12 for a may sense or detect vibrations from the instrument and
bass tone to obtain notes below the regular range of the 60 convert the vibrations to an electrical signal. In operation
guitar, or a setting of +12 for a flute sound above the 704, the electrical signals may be encoded to music control
guitars range.) Clicking the up and down arrows data (e.g., control signal, message or MIDI data). The
changes the transposition in half-step increments. control or MIDI data may include information such as pitch
Quantize Mode Selector 518. Defines how TriplePlay and how long a note is played on the instrument. The MIDI
interprets pitches that “fall between the frets, such as 65 data may include information Such as pitch and how long a
bent notes and reverse bends. (Remember, however, note is played on the instrument. In operation 705, a
that your results are also subject to the settings within receiver, for example, may wirelessly transmit parameters
 US 9,460,695 B2
13 14
for encoding the electrical signals to the encoder. In opera 4. The musical instrument of claim 1, wherein the media
tion 706, the MIDI data may be wirelessly transmitted to a signals include signals for audio, video, or lightening effects.
receiver. The receiver may be coupled to a processor or 5. The musical instrument of claim 1, wherein the second
computer device that synthesizes audio signals. Operations transceiver is to transmit MIDI parameters to the first
706 and 705 may be interchangeable in order, or may occur transceiver.
simultaneously or nearly simultaneously. In operation 708, 6. The musical instrument of claim 1, wherein the encoder
the computer device may output or produce media signals includes memory to store parameters for encoding the
Such as audio signals, video signals, images, or lighting electrical signals to the music control message data.
control messages based on the transmitted MIDI data. The 7. The musical instrument of claim 1, wherein the encoder
computer device may further allow a user to edit MIDI 10 includes controls to select MIDI parameters.
parameters that affect how MIDI data is encoded from 8. The musical instrument of claim 1, wherein the pro
electrical signals generated by the instrument. cessor is comprised in a stomp box, the Stomp box com
One or more processors may be used for processing, prising a synthesizer or sample player and foot Switches for
transmitting, receiving, editing, manipulating, synthesizing controlling or editing MIDI parameters.
or patching digital or analog audio signals. The processor(s) 15 9. The audio or visual system of claim 1 wherein, after a
may be coupled to one or more memory devices. Computers predetermined period of time has elapsed without the first
may include one or more controllers or processors, respec wireless transmitter transmitting a message to the second
tively, for executing operations and one or more memory wireless receiver, the first wireless transceiver transmits a
units, respectively, for storing data and/or instructions (e.g., message that does not include any music control message
Software) executable by a processor. The processors may data to the second wireless transceiver.
include, for example, a central processing unit (CPU), a 10. The audio or visual system of claim 9 wherein the
digital signal processor (DSP), a microprocessor, a control predetermined period of time is based on a maximum
ler, a chip, a microchip, an integrated circuit (IC), or any amount of transmission delay between a command input at
other Suitable multi-purpose or specific processor or con the second wireless receiver and receipt of the command by
troller. Memory units may include, for example, a random 25 the first wireless transceiver that ensures a player of the
access memory (RAM), a dynamic RAM (DRAM), a flash instrument does not experience a delay in response to the
memory, a volatile memory, a non-volatile memory, a cache command when playing.
memory, a buffer, a short term memory unit, a long term 11. A method, comprising:
memory unit, or other Suitable memory units or storage generating vibrational signals by a musical instrument;
units. Computers may include one or more input devices, for 30 translating, by a pickup positioned on, partially within, or
receiving input from a user or agent (e.g., via a pointing within a body of the musical instrument, the vibrational
device, click-wheel or mouse, keys, touch screen, recorder/ signals into electrical signals that are indicative of the
microphone, other input components) and output devices for vibrational signals;
displaying data to a customer and agent, respectively. encoding, by an encoder positioned on, partially within,
In additional embodiments, the present technology may 35 or within the body of the musical instrument and
be directed to non-transitory computer readable storage coupled to the pickup, the electrical signals to music
mediums that include a computer program embodied control message data prior to transmission of the music
thereon. In some embodiments, the computer program may control data from the instrument; and
be executable by a processor in a computing system to wirelessly transmitting, by a first wireless transceiver
perform the methods described herein. 40 positioned on, partially within, or within the body of
What is claimed is: the musical instrument the musical instrument and
1. A musical instrument, comprising: coupled to the encoder to transmit the music control
a pickup positioned on, partially within, or within a body message data from the encoder to a second wireless
of the musical instrument, the pickup configured to transceiver that is not positioned on, partially within, or
receive vibrational signals generated by the instrument 45 within the body of the instrument, wherein the media
and translate the vibrational signals into electrical sig signals are capable of being produced based on the
nals that are indicative of the vibrational signals; music control message data.
an encoder positioned on, partially within, or within the 12. The method of claim 11, wherein the music control
body of the musical instrument and coupled to the message data conforms to a MIDI data format.
pickup to encode the electrical signals received by the 50 13. The method of claim 11, comprising editing, by a
pickup into music control message data prior to trans processor, MIDI parameters for encoding the electrical
mission of the music control data from the instrument; signals to MIDI data.
and 14. The method of claim 11, comprising generating elec
a first wireless transceiver positioned on, partially within, trical signals by the musical instrument, wherein each string
or within the body of the musical instrument and 55 on the musical instrument generates separate electrical sig
coupled to the encoder to wirelessly transmit the music nals.
control message data to a second wireless transceiver 15. The method of claim 11, comprising storing, in
that is not positioned on, partially within, or within the memory coupled to the encoder, one or more sets of MIDI
body of the musical instrument, wherein the media parameters for encoding the electrical signals to MIDI data.
signals are capable of being produced based on the 60 16. The method of claim 11, wherein the instrument is an
music control message data. electric guitar.
2. The musical instrument of claim 1, wherein the music 17. A system, comprising:
control message data conforms to a Music Instrument Digi a MIDI encoder coupled to a pickup, the MIDI encoder
tal Interface (MIDI) format. and the pickup positioned on, partially within, or within
3. The musical instrument of claim 1, comprising a pickup 65 a body of an electric musical instrument, the pickup
device to detect sound created by the instrument and convert configured to translate vibrational signals generated by
the Sound to electrical signals encoded by the encoder. the electric musical instrument into electrical signals
 US 9,460,695 B2
15 16
and the MIDI encoder configured to encode the elec
trical signals into MIDI data prior to transmission of the
MIDI data from the electrical music instrument; and
a first wireless transceiver positioned on, partially within,
or within the body of the electric music instrument and
coupled to the MIDI encoder to receive MIDI data from
the MIDI encoder and wirelessly transmit the MIDI
data to a second wireless transceiver that is not posi
tioned on, partially within, or within body of the
electric musical instrument, wherein the second wire 10
less transceiver is configured to transmit MIDI param
eters to the first wireless transceiver.
18. The system of claim 17, wherein the second wireless
transceiver is coupled to a computing system, comprising a
processor and memory, configured to display a visualization 15
of MIDI parameters associated with the received MIDI data.
19. The system of claim 18, wherein the computing
system is configured to synthesize the MIDI data into media
signals.
20. The system of claim 18, wherein the computing
system is configured to allow editing of the MIDI param
eters.