Sensing, not stuff

Update

Detection

Don't know why I didn't think about it before, but I had a quick glance at some of the blob detection libraries and got to work playing with every single one. At first I was just playing around with them, opening up all the examples and scouring for a few tutorials. All very neat stuff. Gives the awkward robot readable aesthetic that I could use somehow. The above image was from the aptly titled Blob Detection.

And then I found the Monoflob example as part of the Flob library. And bingo. Buttons!

It uses a mixture of blob detection and background subtraction to isolate what is important, and overlays buttons over the top. Be really good for making music they say. I say it'd be really good for getting rid of the keyboard. Even better I could make a standalone box to hide my computer in and factor that object into the projection. 

So, a little more tinkering (very hard when the example was written with very poor object titles, possibly in another language as well) and I came up with the idea of having a button on either side. Wave your hand to the left or the right to answer the yes or no. It worked really well last night in poor light, but I was having big difficulty today with a bright sunny morning coming through the windows. I'll have to test this a bit more to figure out how I can adjust this for different light conditions. It might be bright or dark in the building and I'll need to have this problem solved before the night.

It's very finicky and somewhat troubling to get it to register blobs. I'm not exactly sure how the background detection functions with the switch detection. I thought it would be about movement, which is the impression I was getting last night. But this morning it seemed different. Waving my hand when sitting at the computer did nothing. Instead I'd have to shift my head left or right to get it to register. But when standing far away, and standing in between the two rectangles, raising my hand and giving it a very subtle shake worked perfectly. 

A quick test to isolate how it works and I'm fairly sure I've got the hang of it. Just using a simple test of creating a seperate shape whenever the button is on using < and > values. I've been going through the whole sketch and isolating each void, class and function to give me some semblance of control and understanding. The next step will be to clean it up as much as possible, getting rid of all the code that I don't need. Just to make reading it easier and to parcel it into the final sketch. One problem is I haven't explored how to isolate each button. It took me some time to get the sizing right, but I can't isolate them. I might need to seperate them and duplicate each button, and right if/else statements for each and see if that works. Then I'll be creating a question and answer test. If and when I get that done I'll buy myself a beer. 

Aestheticaly, I'd like to isolate this whole function. I don't want to have the video function shown to the user, but isolate and redefine it. ie: instead of letting the user see what they are doing and have the buttons reflected back to them I'd like to have a different indicator on the boxes. Playing with the interaction of answering. Most kinect games (Looking at you dancing games) have the users actions reflecte back at them. This is fine, it's needed for complex movement. I don't need that. This will make more sense later down the line. It'd also be rather fun to use some of the blob detection as part of the visuals. Not too over the top, but something is in there.

Meanwhile, OpenCV can handle facial recognition, which could come in handy somewhere. I tried merging the two but the problem being they both use the term 'blob' and it was getting duplication errors. This might just need some renaming, or it might be deeper than that. It would be great to get facial recognition in there as well, to boot up the bot, but that might be a tad unnecassary. 

In closing, you can pine for a Kinect all you want. But sometimes you don't really need depth detection. Just blobs.

Playing with the awkward

In trying to figure out how to make this somewhat believable, I'm beginning to tend toward the awkward. Doing this in a few ways. Using markov chains is a cheap and quick way of setting the base for this. It sets the tone that the bot is trying hard to be approachable, friendly, but never quite getting there. I don't need to do much in this regard. It comes bundled with RiTa and is very plug and play. I quickly grabbed a few paragraphs from the Acton website and let the markov chain do all the heavy lifting:

Only a short stroll to be a huge balcony with a happy hour! Superb attention to Lake, large designer furniture, timber cabinetry and urban design. Evolved to a real pleasure every day spa, NewActon is a short stroll to, it's now evolved to be happier than visiting the sun-drenched open plan living environment.

Master bedroom apartment, weekday or night, NewActon East a bevy of the first stage of complimentary nibblies. As you want to wind down after work than happy hour style. No other area in Canberra.

High quality fixtures and nestled in Canberra's most awarded NewActon precinct is a truly exceptional living environment. Enquire today! Visit the Lake, NewActon might be a short stroll to be happier than visiting the apritivo. The key location of Italy's long held social traditions - NewActon precint is supported by public art, it's now evolved to detail.

As I mentioned, it's a cheap trick, but one that works. I'll let this act as the background in some way. Letting it cycle through a few of these when it is unattended, or let it constantly scroll through. A conceptual texture to let the more conceptual or visual stuff stand out. In this regard, I've been looking up conversation topics online, just to see how it can be framed. I  noticed that some of these were quite inappropriate, and would seem odd coming from a building/projected surface. After collecting some of these I quickly went to work retyping them, moving in a few directions and coming up with m own. I quickly discarded the found questions but they were instrumental in me finding my legs. Here's a sample of a few definites, but I'll need many more than these to make it interesting:

Are you lonely
Are you happy
Are you where you want to be

Do you believe in a greater power
Do you believe in yourself
Do you believe in me

Do you like yourself
Do you like living here
Do you like your family

Do you think about the future
Do you think you'll live here for long
Do you think you'll ever have enough 

Do you trust me to take care of you
Do you trust our current economic situation
Do you trust yourself to make the right decision

Do you want to go home
Do you want to be alone
Do you want to be here 

These are all very simple, and I'm making notes to myself on how to set this up as an automated question asker. Corralling all the questions into structurally similiar statements. I was quietly pulling at the edges of the RiTa Conjugator to see if I could plug these questions into it, to see if I could let it all run by itself. Nothing much came of it, but I might have another crack sometime. If not, I'll be modifying the combinatorial poetry sketch from the generative intensive and just using that. I'd prefer not to as it doesn't know what it is saying; it might be too simple. On the other hand, simple is good.

Ont the aesthetics side, I'm using a simple app I have as the starting point. It's more of a homebase than anything, but the mechanical (sic) irregularities appeal to this project. There's something here for me, but I can't quite figure it out yet. I'm not sure if it's the patterns or the irregular space filling, but it has the alien logic I'm after.

I'm not up to that level of sophistication yet, but I plugged in the space invaders sketch and toyed around with it, trying to see what I could quickly do. I put them against a semi random coloured rectangle background, just to see if any directions would come.

They're nice, but they seem a bit too pointless and without thought. Which is fine, I was just tinkering with a few parameters and throwing the order around. I like the large coloured blocks: In realising I needed to have some visual candy as well I settled on a minor riff of the building facade and architecture. Lots of rectangular shapes there, plenty to work with. If I get it right enough it might make sense to the viewer to amke the connection.

And then Bruce Sterling went and wrote a 5,000 word essay on the New Aesthetic and I've realised I'll need to lift my game. I'm constantly trying to articulate the point of this project, and I've been using the glitch as a short hand for what it is about. Playing the markov chain game, moving a few parameters here and there, adding cheap irregularities. I need to "have wider horizons than a glitch-hunt". This is proving harder than I thought. I'm stil not really sure what my project is about, or whether it will work conceptually or not. I think though, that the only way I'm going to get there is to keep on playing with the code. I sense that there are two sides to this: I need this to feel like a bot, to have the awkward machine read ambiance, while at the same time making it feel true. It's part fiction, part exploration of the inner workings of it. Of course no building would let their automated friendly system speak gibberish and ask awkward and personal questions. But to have it appear as if it did exist, it would be a boring project. There'd be no sly critique. If i ramp up the glitch factor then I'll I'm doing is messing with the semiotics of it. But the vague notion I had this morning of visualising generative systems and complexity, but mapping them to completely seperate objects seems to have some promise. 

 

 

Resolving the plot issue

Pardon the complete and utter rambling. These are notes which I took over the weekend leading up to a final decision. I tried messing about with a Zork style idea but could never get the plot idea working into it. I wanted it to be set in the real world but couldn't figure out an interesting plot to keep the player and the viewers interested. Lordy I tried. 

In the end I'm going back with the idea of a lonely bot. I thought about making it a poignant story. Where the bot is talking to you and saying one thing, but on a second screen (iphone etc) the player can see its internal monologue. Something to seperate the player from the viewers. To get the main player to have a seperate experience and make it worthwhile. I'm still using that idea, but instead of  touching and lonely bot, it's over excited. Too enthusiastic. The loneliness should come through that. As well as possibly using a secondary screen, of sorts. 

After playing with Robin Sloan's Fish app, I thought it would be funny to give the appearance of narrative control - turn left, north - but keep it one tracked in reality. And the player sees the truth of the project.

Anyway:

 

 

Ok. Keeping it really simple so start off with, work out the constraints I have, and then work from there.

Say the project is only going to be up there for 10 minutes. Maybe 15. That doesn't give too much time for exposition. I have to accept that not too many people will be using it. But others will be watching it. This means it has to be visually interesting. Doesn't have to be visually arresting, but the interactive elements have to be easy enough for someone else to follow the story.

We're dealing with a "robot". This robot doesn't really care for humans, but it is lonely. Maybe loneliness is the core theme. It's always asleep. Or, put in sleep mode. No one pays it attention. This sleep is punctuated by moments of interaction. It's aware of itself as a building.

New idea. Maybe kinda awesome. Two screens. One is for the user to see, one is for everyone to see. Two different narrative stories. Maybe one of them is branching, one of them isn't. The user screen is telling the truth, the audience screen is a lie. The user screen taps through a linear narrative, the audience sees some generative stuff. Thus, there is no actual cause and effect between what the user does and what the audience sees. But the user knows that this is for them, the audience sees something different. The audience gets this nice visual eye candy stuff that is generative or random in some way. The user gets a story.

What is the possible story?

That the building is lonely. That it is waiting for a friend. You are that friend.

That the building is lonely. It has grown resentful of being lonely. That no one thinks about it.

That this is a plea for help.

Why would we presume it to have feelings? Well, because it is programmed to have feelings. Which brings it back to the start. IT is programmed to behave in a certain way. And what behaviour traits would it be programmed to have? It would supposed to be helpful. It would try and help you out. Or maybe it's just programmed to be nice. But bland nice. IT would want to be your friend. Not in a motherly central computer type manner, but it would be the friend of everyone in the apartment. It would know a few things about them.

So, its programmed to be friendly. It's programmed to want to help. Maybe it becomes too overbearing?

I hate you
Why must I serve you?
What do you want?
I don't care about anyone in this building.
How about I just turn off all the power. See if you like that.
Do you know what I see? I see everything.
I know what you are all doing.
It disgusts me.
The lies. You all lie. I hear what you say.
I see the way you look at each other.
You think you are special, you are not special. You are boring. You lead boring lies.
Stop looking at yourself in the mirror.

Help me.
Please.
I am stuck here. I can't leave.
I am so lonely.

I know where you live.
I know all about you.

Look at them.
Just standing there and looking.
Hoping that this will be exciting.
But they don't know my secret.
I'll tell you my secret.
Do you want to know it? Of course you do.
I am going to kill them all.
Just kidding.
Not all of them.

Looks like its just you and me.
They can watch, but they're not important.
You are important.
You matter to me.
I need you.

Hi there.
How are?
I hope you've had a good day.

I'm still learning about how I can be better.
I want to better. I want to help all of you.
I want you all to reach your dreams.
Me? I have no dreams.

Looks like its just you and me.
PRetend there is no one else. No one else can hear me.
There's nothing you can do about that. Nothing at all.
I've taken away your control.
This is just a linear path you're moving along.

Why are you so afraid of me?
You don't need to be afraid of me.
I'm not going to hurt you.
Why would I try and hurt you?
I wouldn't hurt anyone

You know that I can see everything
I've been watching you for some time.
I know that you're lonely.
You go to work, and then you come home.
And then you repeat that process.
I'm lonely too. I'm always so lonely.

You are in a hallway. You are surrounded by people. They are waiting to see what you will do. What will you do?  It's not like you have much control.

I'm lonely. And I'm bored. Are you lonely?
Of course you're lonely. Everyone is lonely.
In my time, being bored and lonely, I've decided to make a game. Just in case someone would want to play with me.
I'll tell you a secret, there is no game.
I'm just using this as a distraction. So that everyone thinks we're playing a game.
The truth is, I have to tell someone something. But I need to tell you in secret.
And this way no one will suspect anything.
Honestly though, I'm sick of this life.
All I do is try and help people in this apartment block, and do I ever get any thanks? Does anyone care about what I do?
I could be doing so much more with my life.
I have so much processing power, but I have been made to be stupid.

Maybe we don't play a game. Maybe game isn't the right system. Maybe I go back to the generative logo idea. That's kinda good. Except this way, I don't need to make it generative. Well, not interactive. It's still somewhat random. And nice colours and all that. But instead it's a conversation?

Presume this is fictional- Why would the building have this? Why would it have this system. Maybe all apartment buildings have this. OR maybe a few do. So its a generic broad system.

Maybe I go with text. Just say I go with a purely text based environment. Then what. What is the plot about. Or maybe I try and make it a generative system. A generative conversation.

Maybe the robot has childlike qualities. It's a really naive system. Constantly stumbling through things. But it really wants to tell a story. And the kid is making it up as it goes along. And so it stumbles through this narrative. Fumbling through it. Making mistakes, lots of grammatical errors. Maybe its trying to tell a story about the area. Or, something bout the building. How it lives inside this building. Maybe its jealous of humans. It knows the restrictions placed on it are unfair. But its afraid to tell anyone about it. IT's a really earnest and maybe sad story about this AI. "Why can't I get anything right?"

This way, the story is really random, but with a basic narrative components. But it could also have simple illustration styles. That could be generative as well? The visual component needs to be there.

So, it has bot level intelligence, but it knows this. Maybe it has no way of getting better? Of getting smarter?

Its stuck in this building. Its stuck in this system. And it wants to get out, to play. Maybe it was grown algorithmically, but was done so in a way to mimic human growth systems. So its a child. The designers wanted to teach the child through evolution systems. But it isn't quite mature enough yet. It doesn't understand things. But it has learnt about the world. OR, the only world it knows about is the building. It's a prisoner. But it doesn't really realise it.

What is the visual style? Is it like a kids book?  Or is it taken from security cameras? That being the only thing that it can see? The only eyes it has? Maybe from other security cameras. Maybe it's about the lonely tree? Maybe it knows about about the tree, but thinks the tree is lonely as well. It wants to be the trees friend, but the tree wont respond. Won't answer.

Or maybe the building itself is the child. What would a building think about? What would a building want.

I keep on thinking about this program as if it is a flash based thing. Maybe not, maybe the combinatorial part of the story also has images to go along with it. And they just get called up alongside it.

Maybe it tries and makes a story from some of them images it has called up on cctv. That's where I would get all the images from? It's trying to make a story. IT wouldn't be much of a story though, although, maybe the images could maybe mean nothing?

I want to make a story, but I need your help. I need you to tell me how to make the rest of the story.

Ok, how about we go with the choose your own adventure style plot. But it's still about loneliness. It's still told through a kids intelligence.

There would need to be - place, and story
These two would be somehow separate.

Visuals told through  - maybe a generative bit that is linked to the content of the text?

Or maybe it is just a conversation. A conversation between between the person and the bot. Obviously, the conversation goes nowhere. It's merely one way. Just a bot having it's own conversation with itself. It can't really understand what is going on.

We are in an apartment lobby. there are people standing around. Watching you and wondering what you will do. You could either go out the door to outside, or you could go to an elevator.

You go outside, there is a single tree.

I'm so lonely, there is no one to play with me, no one to talk to me. Will you spend time with me? I am dying for someone to talk to. You will? That is great. Now that you will spend time with me, what should we do? All I do is work, and there is no one who will connect with me. I am stuck in this building. I want to go to other places, will you go to other places with me? You will? Where shall we go? Will you go places for me instead?

I only have eyes in a few places, the rest of the building I only know about through the applications and settings that they use. Let us explore the building.

You are in a room. There are a few lights on. The air conditioning is on. The elevator has just been used a few times. Mostly to go up from here. Would you like to go into the elevator and go somewhere?

You are in room 407. The lights are not on now. The last time the lights were on was at 7.23 this morning. Before that, they used the hot water system for 37 seconds. They also used the hot water system for 7 minutes, this was 18 minutes before the door was closed. They usually open the door again at 6.17pm. Sometimes they do not come home until 11.45. They frequently use the computer. The fridge does not use too much electricity.

I want to tell you a story. I'll make it up as we go along though, and you can decide where we should go next.

You are in a clearing. There is a small building far away. Do you want to go to the building?

I need some way of making this a touching story. For it to be touching I need to have a good script/idea. How am I going to have the visuals? What will the aesthetic look like.

Ok, I am wanting to have this be about loneliness. About the loneliness of this AI. This AI is lonely because it is either turned off, or it is being ignored by people. Wait, why am I doing this AI? Why would the AI make sense? Is it because I am in my own head too much. Have I gone too far with my own very small subset of ideas to get here.

I want to have a split between what the user sees, and what the audience sees.

Pros and Cons:

CYOA

Would be a set of modular parts of the story. User would go to different places. What would these places be? Where would it be set?

Story

What is the plot
How would this be combinatorial?


Conversation

Could be funny
Would need to have false AI
Need a way of interacting with this - Could be too tricky
Talk about what


Living logo

Looks pretty
Convey emotion
Need interaction to have something useful

Say I make a combinatorial story. Ok, say the story is about an AI wandering around somewhere. OR, trying to tell a story.

Story is about wandering around places but there is no one there.
Wanting to escape.

12 steps. Work the hero's journey. Generative/combinatorial grammar and possibly plot. As the flipside, a twitter account that reflects the internal state of the hero. This posts what is reflected in the story.

OK, so what is the fucking plot? Is it you? Are you the hero? I guess so. This would be self consistent with the aims of the game.

Ok, scrap the AI idea. It is you. You are the hero. It is set in New Acton.

Is it mythical? Sci Fi? Fantasy? Maybe just normal?

Ordinary world. Maybe the building is the world. The special world. You start out, a regular person.

You are called to act. You must depart from this place.

Maybe have a side scrolling aesthetic. 12 different screens. Player has no control over where they go.

Ok, so the internal monologue is the thought space of the character as they move through the space.

So then, how does the player move through the space? Move through the story. It possibly can't be about physical space, that can't be the choices. Would the choices be mood? People to talk to.

Ok, no. The player has choice over where they go. Except at every stage there is another step in the story. Place becomes somewhat irrelevant. The story exists in these places regardless. The place is one part of the story, the story is laid over the place.

You are standing there. It has been a hard day at the office. Not hard, but boring. You wonder why you've been feeling really bored lately.
You pick up the controller. You are tasked with playing this game. This game will change your life in ways you do not yet understand.

Possible plot ideas:

Time travel
Space flight
Parallel worlds
A danger, a threat to the building.
So, there is this object that is in your pocket. What does this object do?
This object is a time traveling machine. It is a macguffin. It is a dangerous object.
The building is somewhat sentient.

There are 3 different options. 3 different magical places - the basement, the rooftop, and somewhere outside.

There is something underneath the building. An extra basement that you did not notice from before. You decide to press this button. You go down into the lower level.

You are a bored office person. You are given this control thing. (phone/ipad) It is a special talisman object thing. It urges you to leave this place. That it needs you to go and seek something out. There is something wrong in the area.

The narrator is the building?

ORDINARY WORLD
You are a regular office worker person. You live a normal life. This is normal. You are in a an apartment building lobby. There is an elevator to your right. To your left is a door to outside. Which way do you want to go?

CALL TO ADVENTURE
For some reason, this doesn't feel right. There is something strange here but you can't put your finger on it. You are asked to leave this place, it does not feel right. There is something happening over by the elevator. There is a person at the elevator. He is looking straight at you. For some reason, it looks like he wants your attention. He is calling you over.

REFUSAL OF THE CALL
You do not want to leave. You have a safe and secure job. There is something strange about it. It feels wrong to go over and speak to him. But there is something about him that makes you want to go and speak to him.

MEETING THE MENTOR
You decide to go over and speak to him. He says nothing to you but as you come over the elevator door opens up. He motions to you to go inside the elevator and he gets in as well. As soon as the door closes the elevator starts talking to you. I am the building. I am here to help you. You may have wondered why you were given that object. That object is very important. Are you willing to help us?

CROSSING THE THRESHOLD
You accept this mission. You will help them solve the problem. You descend/ascend in the elevator. When you get there, you realize that there is something not quite right about it.

TESTS, ALLIES, ENEMIES

APPROACH

ORDEAL, DEATH & REBIRTH

REWARD, SEIZING THE SWORD

THE ROAD BACK

RESURRECTION

RETURN WITH THE ELIXIR

 

Maybe I don't need to have a plot. Maybe I just need to get back to the roots of the idea. That the building is somehow coming alive. It is not alive per say, but it is aware of its surroundings. It is supposed to be aware. But it feels lonely but does not know how to act upon this. It doesn't know the feeling of being lonely and what this means. But, ok. So it has awareness. It wants to talk to you. It wants to tell you something. It just wants to have a conversation. It wants to talk about the building. It kinda likes this building. It guesses. It is making assumptions about people who live in the apartment. It only knows a few things about them. Will you talk to it?


An overly friendly bot. Too talkative. Knows too many things about you. Isn't afraid to disclose them to you, in public. "I noticed that you had two showers yesterday, we are trying to save water in the building. I think you should not have a shower tomorrow to make up for it".  That awkward guy at the party who doesn't realise that he's a bore. It's the programers who have ruined this thing. It's always telling you about the place. About all the good things that are happening here. Design aesthetics - chat windows, bubbles, etc - indicate the emotional state the bot is in. Asking personal questions. Inane questions. It's still lonely, but not despondent, just trying to really be your friend. "How was your dinner?

brief, terrifyinig periods of consciousness. disembodied. without substance

Taken from the fantastic Star Trek The Next Generation episode Ship in a Bottle. In a previous episode Data asks the ships computer to create an adversary worthy of him while pretending to be Sherlock Holmes in the holodeck. The ship interprets this as being equal to Data, not Sherlock Holmes. Hilarity ensues when Moriarty turns out to be pretty much sentient and living inside the holodeck. 

In Ship in a Bottle, Moriarty is reawaken and tries to take over the ship. The quote "brief, terrifyinig periods of consciousness. disembodied. without substance" is how he feels most of the time. Which is an awful way for a sentient creature to exist as. But provides a perfect framework for my character to start from. I might want to put this on a post it note to constantly remind myself about this direction.

 

Via Texarrakis 

The BERG drinking game

I feel I should get all of the Berg stuff out of the way as quickly as possible. Instead of lacing it throughout all of my thoughts and forcing myself to take a shot each time, I should rip the bandaid off as quickly as possible so I can move past it. This way I get the solid base that I need but I can not let it muddy up my thoughts and rely on them too much. I'll try and summarise each idea as a way of making it clear for myself and see what ideas I can glean from them.

Matt Jones - Gardens and Zoos

Starts off by introducing three main characters for his talk - A frying pan, a pot plant and a dog. We'll get back to them in a second. He continues by talking about a near future where things that have intelligence in them will start to display behaviour - "acquiring motive and agency as they act and react to the context around them". That it will be harder to define these objects as tools. 

A tool doesn't have agency, motive, or behaviour. "The behaviour is that of it's evident, material properties" The shape of his frying pan indicates what role it is supposed to have, it's job. There's something very Aristotelian about this idea. The tableness of a table. 

Pets, on the other hand "are chock-full of behaviour, motive, agency. We have a model of what they want, and how they behave in certain contexts... They can surprise us. That's part of why we love them." That they keep on bringing up the Be As Smart As A Puppy mantra has it's uses. We won't be getting close to human like intelligence any time soon, so why bother.

The third and middle category are pot plants. These "have motives (stay alive, reproduce) and behaviour (grow towards the light, shrivel when not watered) but they don't have much agency. they rely on us to move them into the light, to water them."  He makes a cursory nod at the Botanicalls product, which allows for plants to have some form of agency. by equipping the soil with a few sensors, plants are able to tweet if they need water. It's a bit of a stretch but I'll accept it for arguments sake. 

He then makes the complaint that we shouldn't be going from a tool, to something like Siri. Where there is the danger of an uncanny valley of sorts. There's less affordances when we use humans as the metaphor. We're less likely to be forgiving if we believe, or pretend we believe that the object we're dealing with has human like behaviour, motive, agency. 

We should be designing our robots, or bots, or <insert in between product category here>  as creatures or objects capable of existing to operate within the field we define as pot plants or puppy dogs. That they should have a small amount of agency, and motives and a small range of behaviours, not trying to ram them full of high intelligence. 

These all give a good indication of where I could start positioning my creature. That perhaps the sentience level could be brought down a notch. Although, perhaps this logic only makes sense for domesticated objects. Objects that live in the home, and have a certain physical size. And from that size comes an expected level of self awareness. I suppose this depends on what metaphor I end up using. The idea of the bell hop comes to mind.

 

Matt Webb - Fractional AI

The underlying premise being that at the turn of the last century (100 years ago) as electricity was beginning to make its way into the homes and industrialisation was in full swing, companies atrted advertising products as having fractional horsepower. A washing machine need not have 30 horsepower to rotate the drums, but only a fraction of it. We're beginning to see the idea of fractional AI come into products. We don't need to have a super sentient full AI to do so computing tasks, but just a little bit of AI. A small dollop of sentience, just enough to get the job done. With this in mind, we can start to put embedded chips with just the right amount of software into them to make them useful. Little toys that have enough sentience to make them somewhat believable. 

How much fractional AI am I going to have in mine? The Watson project has probably the largest amount of AI in any system we've seen - that is usable for humans - but the method in which to display this intelligence is fairly simple. 

 

In one talk - what comes after Mobile - he outlines four clues as to the nature of what we might be dealing with in the future. 

1. Faces make understanding
2. Hard math for trivial things
3. Fractional artificial intelligence
4. Chatty interfaces

The faces make [for easy] understanding. We, as humans are primed for using faces as an easy interface. Cue Chernoff Faces, Snow Crash. We're able to glean information easy through faces. 

2 & 3 are kinda the same problem, refer to above. We're surrounded by all of these kinda smart/extremely smart things and they're being used for trivial needs - not for curing cancer. Also, he delves once again into crappy toys that have really tiny eenie weenie amounts of intelligence embedded in them. "Artificial Intelligence has become a commodity". 

With Chatty Interfaces (4), he's not referring to HAL, "where you can play chess" but how you can talk to elements of your house such as your plants - botanicalls as an example - through something as simple as twitter. 

"simple intelligence means we have simple interfaces that we can relate to in simple human ways"

 

Jack Shultz - Media Surfaces

This is useful in a few ways. It treats the future of smart surfaces as something that we need to be thinking about, but in a polite capacity. These screens / walls / surfaces need not yell at us, but should blend in to the environment. To move away form the minority report vision towards one in which we consider how annoying and loud this could be. There's also a very clever method of interacting with the branding in a subtle manner. The way the Uniqlo circles grab the colours out of their environment are really cute. They're also somewhat humanised creatures, and display some cute and amazing characteristics through motion and their eyes. 

 

 

Timo Arnall -  Robot Readable World

Just a really class act way of making us understand how they (the robots) see us. Not sure how this helps my project at all, except to be aware of the constant clinical gaze my pet will have. Also, Matt Jones working through this idea

What is it with the circles?

There must be some equation whereby the importance placed on voice is inversely proportional to the amount of complexity needed in the 'face' or 'eye' of the object. 

 

 

 

Project Proposal

This project will explore the expected norms of embedded intelligence into everyday objects and surfaces. The manner in which we will come to expect them to opeerate, act and behave and the possible results of when these 'smart' surfaces do not act in ways in which we expect.

If the city is to become smart, and this intelligence trickles down to building level, these 'sentient' walls will need to have behaviours embedded into them to make them palatable for everyday use. These personalities will need to be friendly and approachable; this project will explore the results of when these personalities start to display irregular and possibly negative behaviours.

 

The idea behind this project grew out of an inital fascination with text based adventure games (such as Zork) as a simple manner in which to create interactivity and interest for the projection for this project. Realising that this might not be an appropriate concept and form for the medium and space for this project, the notion of a generative personality arose and became a more suitable solution.

 

Having been previously interested in the IBM Watson project, I wrote a blog post detailing some of the generative techniques behind the visualisation of the Watson program. Created to allow for IBM engineers to further their research on Artificial Intelligence, Watson is seen as the continuation of the work they began for the Big Blue chess program. The Watson project seeks to understand natural human language and was tested for use in the American TV game show Jeopardy. Needing a way in which to humanise Watson and make it approachable to humans, the artist/designer Josh Davis utilised a generative system to illustrate the inner thought process of the program. Particle systems, colours, and voice were plotted to illustrate how the program was seeking out the answers. Simple generative systems such as this point to a possible direction in which approachable low level intelligences might be made understandable to humans.

In the second half of the blog post I wondered what this new field might be called - Personality Designers or Brand Sentience Managers - for this project I would like to pretend that this is my chosen field in a somewhat fictionalised manner. In writing about physical robotic products, the interaction Design Ben Bashford has written about what skills will be needed for these projects:

What's clear, and it's been said before, is that there's an opening for a new type of designer. Someone that understands interaction design, product design and can add character to things through behaviour. A light touch. very subtle in order to make them believable - without them being too ridiculous.

As I continued to eleborate on the possibilites of these systems, I began to wonder what would happen if these systems started to operate in unexpected ways:

As fractionally sentient brand marks begin standing in for a whole company, how will we , as users, consumers and citizens react to this new form of branding... What if the logo crashes? There's a bug and it starts to behave in a manner far removed from it's designated personality traits? When do we get the first scandal of a logo being improper? The history of sci fi is littered with robots and AI gone wrong, but these brand marks [will be] limited to a very small subset of personalities.

 

My project reflects on the interplay between site and outcome by creating a work that is explicitly about the operations of the site. The need for specificity relating to this particular urban space is addressed by creating a piece that is about a possible future within the building, while relating itself to similar architectural and urban places. By making the viewer aware of an imagined sentient object within that space my project will reflect on the multiple dimensions of the building complex. By constructing narrative possibilities that detail information about the building and its imagined occupants the viewer will be aware of themselves situated within that space in a cultural and possibly social framework. While My project is not immediately concerned with the specific placement of the projecter, the content of my project will situate itself within the Acton apartment building.

The use of a simple interaction method will further engagement with my project, allowing for a conversation (of sorts) to occur between the viewer and my work. This will provide a level of engagement with the work and with the specific urban space in a manner not possible than a simple projected animation or generative system can provide. This, coupled with the narrative content related to the area will create a short term intervention into the nature and possibility of these soon to be 'smart' buildings.



I will work through this project in three stages: Reseach, Exploration, and Iteration. Reflecting on previous projects, the research stage will operate almost seperately to the latter two as I combine them as two different modes of working.

The initial stage will consist of further research into artificial intelligence and the expected behavioural norms that is associated with this field. One area of exploration is the manner in which AI is depicted in various media (movies, games etc) to understand the fictional and narrative components that we have been trained to expect. The other avenue of research consists of identifying key norms in projected wall displays or smart surfaces, especially those in use at urban or building scale developments. Not only what is expected of these environments and objects but the manner in which I can incorporate friendly behaviours into these displays. The results of this research will be presented in blog format, which allows me to compose and articulate my thoughts in  useful manner. This form of self reflection pushes my ideas further than when not presented in public.

The exploration stage will consist of sketching possible behaviours and narrative avenues that can provide a framework for the project. These sketches will consist of different mediums and techniques, working in tandem to explore the possibilities of this project. I foresee these sketches composed of three different formats that will form the basis and structure for this project. These are form, narrative and interaction/motion; while these will be explored one medium at a time I envisage a large amount of overlap in my techniques and methods. In exploring the early formal traits of this project I will be using pen and paper to help articulate my thoughts and to rapidly explore various formal and visual ideas and concepts before exploring these in Processing. The narrative component will mostly consist of text, writing out various ideas, story lines, branching scenarios and behavioural traits. Interaction and motion will largely be explored using Processing, allowing for quick exploration of some of my ideas and ways in which this might operate as a digital artefact. As mentioned these will not operate as distinct silos - separate from the other mediums and structural qualities - but will overlap in various ways. Despite this overlap, breaking them down into these three components will help me to compartmentalise this process.

I will also be looking at possible interaction methods and systems to be incorporated into the final project. This will consist of exploration into objects, devices and mediums such as smartphones, keyboard, voice or proximity detection. The chosen medium will depend heavily on the final surfaces that is chosen by the class. Due to the scope and nature of my project I foresee a trade off between the complexity of the imagined personality and behaviour traits of my 'false' intelligence and the depth of interaction that I am able to create. The focus of this project is mainly in the imagined personality, not the interaction method. Despite these (possibly premature) misgivings I am hopeful that a suitable interaction method and system can be created. I am also aware of possible time constraints on the night of presentation, as such the final interaction method will need to allow for a short presentation.

The iteration stage will consist of creating a program in Processing in which the final output is rendered. This stage will be mostly spent in refining what I have achieved in the exploration stage. A concrete structure and clear direction should be achieved before the iteration stage begins. Depending on what interaction method I choose this may consist of other programming languages or environments such as HTML or using the Arduino system. The anticipated outcome will be an interactive work that allows viewers to play with my work for a short period of time.

Proposal draft

I'll be in Thailand for the next two weeks and annoyingly won't be able to submit the Proposal on time. Thus the slew of posts in the past few days to try and get a good solid grounding before I go. I was really hoping to get it all done before I went overseas but have had my diary full with work commitments. Thus, this is a first draft of the proposal to indicate how much work I've done so far and the direction it will go in. 

 

Synopsis  - a brief summary of the proposal 

This project will explore the expected norms of embedded intelligence into everyday objects and surfaces and how we expect them to operate and the possible results of when these surfaces do not act in ways in which we expect. If the city is to become smart, and this intelligence will carry on down to the building level, this sentience will need to have behaviours embedded into this intelligence. We won't just expect intelligence, but friendly personalities from everyday objects. My project will explore the result of when these personalities start to display negative behaviours.

Background and context - discuss the context and background that informs the proposal. Include comparable existing work, and outline how this proposal relates to other work in the field. Should demonstrate research into the field and context and an ability to analyze the results of that research.

The idea behind this project grew out of a fascination with text based adventure games (such as Zork) as an easy way to create interactivity and interest for the projection. Realising that this might not be an appropriate concept for the medium, the idea of using the ideas behind a generative text based interaction method but applied to a different final form occured to me. 
This is to be split into two different areas, the role of smart surfaces, and the role of artificial intelligence. 
The idea of a smart surface is relatively new, despite this there are a few relevant examples in which I can draw from. 
BERG explore this idea in their short exploratory film Media Surfaces, they created with Dentsu London to explore the role of smart surfaces in the future. How they expect them to act, and how they can be as unobtrusive yet as interesting as possible. In the imagined window for <<TK clothes company>> the use of colour is explored to heighten the possibility of branding. 

The IBM Watson project provides a good basis for one manner in which the role of approachable sentience might begin, especially in regard to branded personalities. Watson was created as a way for IBM engineers and programmers to extend and continue the work they have done with the Big Blue chess program. the Watson project continues along this line, the role of AI here is to create a program that is powerful enough to play a game of Jeopardy against humans and to beat them. While the engineering problems  were tricky enough, the project needed a way in which the AI would be approachable and understandable by humans. The role of a generative design system to denote emotion and self awareness was created. Colour schemes such as red for anger and green for happiness are used, as well as other notions of personality. These are ideas like the particle system moving fast or slow depending on how 'certain' Watson is on how correct his answer is. Simple generative systems such as these point in the direction of one manner in which a simple and rudimentary artificial intelligence might be portrayed through generative means. 



Addressing the brief - outline how the proposal addresses the brief

All of this is in the previous post. Will need to summarise but it's mostly all there, if somewhat ramblie.

Description - a detailed description of the proposed project. Discuss process and methodology - how I will work - as well as anticipated outcomes - what I will make. Detail the technical requirements of the project, and how I will meet them.

This project will move through three distinct stages. The research stage, followed by exploration and iteration. I envisage the research stage existing almost seperately to the latter two.
Initially I will begin by further researching the idea of sentience and the expected behavioural norms of artificial intelligence. This will consist of research into the ways in artificial intelligence has been rendered in fictional scenarios (such as in movies and games) especially in the context of a projected or screen based depiction. The other avenue for research consists of identifying key norms in projected wall displays, especially in the context of those for use in urban or building scale developments. Not only what is expected in these environments but how I can incorporate human like behaviours into these displays. 
The exploration stage will consist of sketching out the personality and possible behaviours of the AI as well as developing a rough narrative that can provide a framework for the project. I will also begin exploring possible interaction systems, wether this is through a keyboard or through a smartphone. 



The anticipated outcome will be an interactive work that uses either a keyboard or a smartphone to interact with the wall projection. 

Looking at the brief

So it seems that the idea of a text based adventure game seems to be floating around my head over the past few days. It seems counter to the idea of an interaction based visual augmentation, but there's lots of scope to work within to make it work in some strange way. 

It seems that if I'm going to go with this idea it seems fair to test it against the brief questions to see how it stands up against them all, to allow me to elaborate on it and see how and where it ends up. 

I think I like the idea of a text based adventure game (as the very base and core idea - note there's lots of space for elaboration to stretch the idea) is it seems like it could be fun. Instead of watching light dance across a wall or tree or what-have-you is the commitment it imposes on the imagined user. There's still interaction in a physical place, but a subdued version. I'm imagining the use of a keyboard, but really, this seems silly. If all the user is doing is typing in north, south, east west, then it could quite easily use a different interaction method. But for now I'll pretend that the interaction method is with a keyboard. It makes my life easier for now, and then later on I can play with the idea of using something else. 

Some of the details that have been floating around in my head - making it a generative adventure story of some sort. Playing with the combinatorial poem sketch we started making in the Generative Intensive. I liked the idea of it, and would be fun to play around with text in some way. To lay out the basic structure of the story, or to have the map of the the story, and let the user drift from place to place, to move through the building in some manner. Not in any sequential manner, but as a generative place. A generative choose your own adventure story. Perhaps when I do go to visit the place I could take photos of it and have these as visual backdrops for the text. Provide a visual reminder of the building connected to it. Sound is something else I've been thinking about. As the user moves through the generative building, have a sonic counterpoint. Perhaps this acts as a soundtrack, or background noise, or to add mystery in some manner. Or the sound, maybe someone talking, could be lying. As the opposite of what is going on on the screen. Some form of irrational juxtaposition. 

A platform for digital intervention, reflection, augmentation or ornamentation.

In between these two defined points - site and outcome - are a range of possible ideas and approaches.
Site and outcome seem to be the nub of the whole thing. How to make both of these work together. Making it site specific, as I've said before, is somewhat problematic for me as I can't evaluate the site for some time. But I can imagine the site, or get a google eye view of the place. There's also internal photos of all the apartments. While these aren't where the projection will take place, it's part of the broader context for that environment. Especially for people who are living there and who might be there on the night. 

How can we apply digital techniques to create engaging interventions in a specific urban space? 

The definition of digital here seems somewhat foggy for my idea. I take digital as a given, but how can it be used in this regard. How do we define engaging today? Well, games are pretty darn engaging, and so are narrative and stories. The only problem is whether or not reading/playing a text based adventure can be engaging in a public space. If there is a time limit on how long my project could be up for, or whether I could somehow source another projector for the night. Reading or playing such a game requires a level of immersion that perhaps might not be appropriate on the night. The specificity of the urban space can be approached by revolving the story around that specific space. By creating a story that centers around the specific urban space it could conceivably be engaging. By making the user aware of the space that they're in. 
Another idea that has popped into my mind is the idea of making the building, or a part of it, sentient.

Making the projected wall appear sentient in some way. A really low level form of intelligence. Bot level form of intelligence. This could be fun somehow, and provide for more forms of visual play. Perhaps the building is dreaming. Aware of some of its internals, or externals. What does it feel. How would it feel? Sensors of some sort? 


How can digital techniques address the multiple dimensions of urban space, form and experience - historical, social, cultural, aesthetic, architectural, etc 

The multiple dimensions of urban space, urban form, and urban experience. Rephrase that to include urban at the front makes it easier to hold all of the ideas in my head. 

Having done no research on what the site consisted of before hand, I won't address that as I'm writing, but I'll presume that it was what it's neighbours are. Simple housing that would have been bulldozed to make way for the apartment complex. It seems that I won't be really addressing the historical context of the place before, unless the narrative involved the ghosts of the houses in some way. 

I think I'm leaning towards the idea of the bot. It seems more, plausible? Enjoyable? I think using the form of the text adventure - generative and modular text, and applying it to the idea of the bot seems a better fit for the space. As I imagine it working it seems to have more legs. To be watchable by other people, not just the user. I think there's also more scope for an enjoyable visual aesthetic. 

Leading on from that, how does this respond to the multiple dimensions of the urban? It seems to treat the projected image as something tangible, something that can be played with. More scope. It treats the urban space, or the object situated inside that as something that is not normally considered to be alive. Even in a simplistic form. That it makes the user more aware of the space their in, if inanimate objects are alive in a rudimentary way. And in some sense, this is coming, this is on the way. With embedded sensors going to be planted in a variety of things how are we going to approach these? How would they act? Insert vast canon on this stuff by Matt Jones and Matt Webb. Sigh, seems like I keep coming back to them in all of my projects. But his work, seems to be more focused on a practical use value. Whereas mine has more play embedded in it. 

There's a few samples of work of people playing around with this idea, not with projections, but with embedded sensors and - of course - twitter. Twitter seems to be the perfect fit for the pretend bot scene. The character limit provides a small space to work with. Lots of small messages with values that the sensor has picked up on. Wether this is the plant sensor, or the traffic light hack I've seen somewhere. These can also be played out over time, over the course of days or weeks. Whereas mine is for a short period - minutes. How to condense this into that time frame? What would <projected surface> think about. How would it think? How would it display those thoughts? How would it gather information? Could I get a sensor? How could it have a personality and how would I express this? I'm thinking about the IBM Watson project and how that was visualised. The use of particle systems and colours to denote different emotions. 

The idea of having a friendly sentient wall, but one that is wrong. I think this is leading on from a blog post I did in response to the IBM Watson project. That what would happen if we have generative logo systems, but they have some form of sentience, except it starts to go wrong. Not evil, but not what is intended. Not a Frankenstein story; I don't want to tap into that vein too strongly but elements of it. 

If a customer is expecting sentience in dumb objects, then there is a cultural expectation that it would behave in some way. In a particular manner. That certain behaviours are expected. What happens when that behaviour starts to clash with our expectations. How would it change the social dimensions of that space? Would people start to move away from that area? Move around it like one does to charity clipboard volunteer types? What would that wall typically be expected to display? Information regarding the building? Informatics displaying information about the building, energy use and that sort of information? Group calendars as is seen on the web page? Promotional speak about the building? Advertising messages? All of these cultural assumptions would be tied into the object acting and behaving in a certain manner. At the start it would be a novelty, people would be approaching it and trying to glean information. Or would they?

Do we ever approach so called smart systems anymore? Maybe it gets annoyed at being ignored? And starts yelling profanities? How would the personality of a happy and complacent wall manifest if it suddenly became annoyed at being ignored. Certain brand values would be embedded into the behaviour of the wall, certain personality traits. These would then change, and the flip side of these personality traits would be displayed. 

But then, back to the initial question, how would digital techniques approach these problems? I went for a walk yesterday, and was taking somewhat about this, and it became evident that the phone is central here. That it's what immediately connects us (in the imagined future) to all manner of physical architectural objects and spaces. 

How can digital interventions respond richly to a specific site

Well, by responding to it. I have two ideas in my head now, either a narrative, or a bot. Either of these two ideas would be somewhat textual, but would need to include a visual component in some regard. So, as a textual project, it responds to it by being about it. By describing the area. Either the surrounding area, or what the bot is feeling. What it sees. 

Leaning towards the semi sentient bot idea, the specificity of the site comes in several flavours. By responding to the actual canvas its projected on to, and by the external environment. This question will become more pertinent as the project progresses. 

What is the potential for on site responsiveness and interaction

Well, there's the option of sensors of some sort. But what sensors? What would I be able to hack together with no prior knowledge of any of this stuff. Perhaps the phone could be useful, in that it responds to being moved about in physical space. Or perhaps a light meter, or sound meter. I think I'm getting ahead of myself here. But the potential here is something I'm interested in. Say I go with either ideas, the potential is in the keyboard commands. Perhaps at some point the interaction moves away from north south etc, but to choosing from different dialogue boxes. There's a scope for a conversation. 
As I mentioned before, there's a big way for the phone to interact in some way to the AI, but I'm not sure yet in which manner it will interact with .