Chaos in the Cloudflare Lisbon Office
148 comments
·March 17, 2025nthingtohide
emot
that's a cool experience. it's really mesmerizing and calming watch one or two of these machines, because of the way the waves move, similar to ocean waves. there's this video that shows it at the end: https://youtu.be/mYLvRaMmfho
4gotunameagain
It's amazing, but data sculpture ?
Really ?
Why do people feel the need to appeal to authority for the things they made. This has nothing to do with a sculpture. But it doesn't take anything away from its value.
lnrd
> sculpture, an artistic form in which hard or plastic materials are worked into three-dimensional art objects. - Encyclopaedia Britannica
umvi
Is this just for PR/show and the real entropy generator is standardized across sites to be something boring in some back room? Or if a terrorist came in and cut power to a chaos wall would it really cause an extended outage?
eastdakota
Great thing about entropy is that adding more never hurts. This is one of many sources — both more conventional as well as unconventional — that we use. If it were to go offline, or somehow be corrupted, it wouldn’t hurt our ability to generate entropy across the Cloudflare network.
What I love about this, the lava lamp wall in San Francisco, and the double pendulums in London, is that it takes something very abstract and makes it tangible for our team and our customers.
AdamN
These are my favorite types of marketing - what I'll just call 'part of the actual stack'. Would be great if there was a Berlin office so I could join!
AStonesThrow
Ah, pendulums. They had some cool ones in the museums we used to visit. One was 2+ stories high and it would knock down dominoes to reckon the time of day as the Earth perturbed its motions.
Another much smaller pendulum in the hands-on science exhibits, you scooped sand into it and then set it swinging freely across a square black surface. It would trace out amazing patterns as it spilled sand hourglass-style.
So then some bully would rock up next to me and smack the pendulum, stop it from swinging, and spill a big blob on the formerly-geometric pattern. And they invariably said "just to see what would happen". AStonesThrow would have a small meltdown or become rather indigant. I suppose their empirical science is just as valid as kicking down sandcastles on the beach.
And that's how I came to prefer single-player games...
AStonesThrow
Today I learned a new definition: Lissajous
https://en.wikipedia.org/wiki/Lissajous_curve
The sand pendulum drew lissajous curves as it swung. [I learned it today because they called the ESA Gaia probe's orbit "Lissajous" around the Lagrange point.]
https://www.esa.int/Enabling_Support/Operations/Farewell_Gai...
jstasiak
> Great thing about entropy is that adding more never hurts.
I used to think the same but here's a counter-example of a (hypothetical) attack based on a malicious entropy source being able to manipulate the hash/PRNG output:
https://blog.cr.yp.to/20140205-entropy.html
Now, it's not necessarily the most likely attack to materialize, as already pointed out downthread: https://news.ycombinator.com/item?id=43391377.
crazygringo
It just injects additional randomness into randomness that is already perfectly fine. It's basically for PR/show, but it is actually used. But without it, things work fine too.
thih9
If it silently crashed and started to output a static number, would this affect any systems negatively?
jedberg
I asked this last time about the one in London, and was told that one of the checks is that the image has changed since the last run. Otherwise the data isn't used.
Prevents not only technical issues but attacks like someone blocking the camera or putting a static photo in front of the camera.
crazygringo
Surely not. If you're seeding a PRNG from multiple sources of entropy, you generally concatenate them. Or if you were limited in bytes you'd XOR them.
This is why, in an app, you might seed with timestamp and process ID and /dev/urandom, in case any of them happen to be non-unique or unsupported.
luuio
a random number (existing entropy) XOR with a static number (the crashed wall) is still a random number, me think
ycombinatrix
Probably not, unless it is their only source of entropy.
tptacek
It is 1000x for PR/show. There is no practical security benefit to stuff like this. There's also no meaningful risks to doing it. It just lets them write fun blog posts.
dadrian
If they actually integrate this into randomness on their TLS servers, the only risk is that the system for getting the entropy from the lamps and waves somehow screws up, fails to parse an HTTP request or something, and accidentally seeds the whole system with no entropy. Whereas doing literally nothing and just letting Linux boot correctly on metal would be perfectly secure.
tptacek
Right, but there's no way Cloudflare is making that kind of mistake. If it was a random person on HN talking about how they'd hooked up a bespoke hardware RNG to their TLS stack I'd write some tut-tutting thing about what could go wrong, but here the security of their system collapses down to the LRNG just like every else's.
necovek
In theory, if there is a bug found in the entropy generation used by them and everyone else (like in /dev/urandom), by mixing these in, their source of entropy would still remain unpredictable.
Though this is certainly a pretty expensive if nice looking backup entropy source.
tptacek
I think you'd probably have a hard time finding a cryptography engineer that agrees about this. I think in Cloudflare's case, they have the resources to "do" this "safely" (in reality, I would expect the overwhelming majority of their fleet uses conventional random number sourcing anyways), but a less-equipped team is actually more likely to harm themselves than hurt themselves trying to do it.
It all nets out to "these are fine blog posts; don't try it at home".
swyx
i mean its also nice office decoration and also lets them rank on HN. many companies spend far more on employer branding with far less results.
tptacek
Yes, we think the same thing about this. I don't have a grudge about it, as long as people don't try to build their own versions of it for security purposes.
linsomniac
>if a terrorist came in and cut power to a chaos wall
That's an interesting way of saying "if the wall loses power". Your name wouldn't be Hans Gruber, would it?
6stringmerc
“I’m a thief and an excellent one at that, and considering I’ve just promoted myself to kidnapper, you should show some respect!”
gunian
meh hollywood needs to upgrade its villains lol their narrative is just boring and trite
brookst
It's an odd world when a hypothetical terrorist is looking to reduce entropy.
amarcheschi
It sounds something straight out of a cyberpunk novel, some sort of enthropy thief
praptak
Entropy does not always align with what we perceive as chaos/disorder!
Crystals are generally considered pretty orderly, yet the oversaturated solution actually gains entropy when it crystallizes.
namaria
That's misleading. The entropy of the solution is increased and that's because you're removing solute from it. It has nothing to do with the crystal or its entropy.
thinkingtoilet
If your goal is to weaken encryption, reducing entropy could be a step along that path.
brookst
Sure, but it is amusing to imagine terrorist cells plotting to reduce entropy.
There's a SF story to be had here: the global superintelligence uses bits of litter and fallen leaves and stuff to generate unbreakable encryption; the terrorists wage a global campaign to clean up litter, prune trees, get everything neat and orderly in order to hack the system...
deadbabe
It’s a form of eco-terrorism.
coppsilgold
The image sensor noise will produce more bits of randomness than the chaotic display.
There is also dedicated "TRNG" hardware which will measure random thermal noise. Some will even get fancy with quantum effects.
Any source of randomness will do, you just feed it into a hash function and extract uniform randomness you can use in cryptography.
For example, if you have an image sensor that takes an image (and does no post-processing) and you feed that image into SHA256 you get 256 bits which you can use for cryptography. As long as the image is never saved there is no practical way to recreate the input and in fact the input will contain more entropy (degrees of freedom) than the output, so no one would even want to try. Most of the degrees of freedom in the image would come from sensor noise and not the scene, so you don't even need to take off the cap from the camera.
In practice, multiple sources are combined. The Linux kernel does this for /dev/[u]random though it doesn't use the camera. There is a potential risk with such combination: one of the inputs may come from a source which is able to interrogate all the other sources, it would then be able to adversarially choose its contribution to skew RNG results. This is a somewhat obscure and unlikely threat model.
jgrahamc
Details of how we use this are here: https://blog.cloudflare.com/harnessing-office-chaos/
jgrahamc
That sort of happened on "NCIS", Season 16, Episode 1: https://blog.cloudflare.com/statement-concerning-events-at-g...
bomewish
What’s the second level analysis here? We know it’s not really necessary or helpful for the ostensible reason (far cheaper/more reliable ways of capturing entropy) — so we conclude it’s a marketing gimmick. Yet for the gimmick to work they have to pretend it’s useful. They’re not fooling themselves or anyone else, though.
So what’s really going on?
Is it:
- it IS somehow a good return on investment??
- marketing had a budget and didn’t know how else to spend it, and no one wanted to be the unpleasant person and say how it’s all a silly waste of money?
- they are making a tonne of money and no one really cares, so we’ll just spend it on fun cool stuff as long as there’s a plausibleish story to go with it?
- fits with a broader global company branding concept that leadership seems to like, so there’s just the momentum to keep it going (and see points above)?
I can’t figure it out. I agree it’s cool! Just the make believe puzzles me a little. I’ve not worked at a big corp like this and just have to understand what’s actually happening.
eastdakota
If it were merely marketing spend for customer acquisition, I bet the ROI on the lava lamp wall in SF has been 100,000x. This isn’t hard to figure out.
babelfish
It is a gimmick, but targeted at recruiting, not sales. Cloudflare is known to be “blog driven engineering”
everybodyknows
Not #3: Trailing 12 month P/E is negative.
lysace
Neat and fun PR stunt. Tech needs more of these!
SGI did this almost three decades ago:
https://web.archive.org/web/19971210213248/http://lavarand.s...
...harnessing the power of Lava Lite® lamps to generate truly random numbers since 1996.
According to https://www.lavarand.org/news/lavadiff.html:
Seed production rate was about 8000 bits of seed per second on a 200 MHz SGI O2 under IRIX 6.5.
The patent has since expired: https://patents.google.com/patent/US5732138A
(And Cloudflare re-implemented it, seemingly starting just after the patent expired in 2016.)
eastdakota
Had no idea there was a patent. Even if we had, think we’d have risked it.
lysace
Perhaps someone in the org knew. CEOs don't get all of the details ;).
Nevertheless, it's a great tradition to carry forward and I'm happy you guys are doing it.
kylecazar
It's a cool concept, but I just came to compliment the incredible view from that Lisbon office!
mykowebhn
Judging from the office view, I'd guess it's situated not far from the Museu do Oriente. It's a really nice up-and-coming area, although it's not close to any Metro stations.
Edit: I just googled the location and it's right next to LX Factory, a rehabilitated, trendy shopping area where one of my favorite bookstores is located (Ler Devegar).
rcarmo
Yep. It's pretty nice
brabel
The whole city of Lisbon is amazing. It's very underrated but worth a visit for sure.
lopis
How is Lisbon underrated? It's completely, chocked full of tourists and wealthy expats, and practically devoid of permanent residents because no one else can afford to live there any more.
regularjack
I agree with the sentiment, but it's very far from being devoid of permanent residents. That's maybe true for a couple of downtown neighborhoods, but definitely not true for the city as a whole.
brabel
It's the 64th most visited city in Europe (EDIT: not Europe, world):
https://en.wikipedia.org/wiki/List_of_cities_by_internationa...
I think it deserves to be higher.
epolanski
Underrated under which metric?
I would say it's rated highly by tourists and expats.
Novosell
Underrated basically always means "I really like it but no one in my social circles has brought it up unprompted", in my experience. Really annoys me.
Saw someone call the The Wild Robot, nominated for 3 Oscars, underrated the other day.
bravura
Subtext: In case a) you want to move to Europe, b) cannot tolerate weather that differs from California, c) like Golden Gate bridges, are you aware that Cloudflare has a Lisbon office?
Source: I live in Portugal and notice all the discussion about it from my Bay Area friends who want to expatriate. To here, specifically.
gusmally
Expatriation (immigration?) to Portugal from America furthers Portuguese wealth inequality and housing crisis. Lisbon is already basically a playground for foreigners and if you do decide to move there please try to assimilate by learning the language, etc.
Source: Portuguese/US citizen. Lived there for a year with a "good" Portuguese salary of 2000 euros a month; much different lifestyle than what the typical US tech worker is accustomed to. And not doable long term unless you plan to retire in Portugal and not do much travelling.
mykowebhn
These type of comments make me cringe. Nowhere does the OP describe what Portugal is like in itself. It's very often compared to something else, like California. Portugal is Portugal.
bravura
My comment was about "why would cloudflare make a fluffy blog post like this?" And I believe its to attract people to Lisbon.
I never said that I support this oversimplification of Portugal. Which is why I live in a small town, socialize with Portuguese, and study Portuguese.
collingreen
Visited Lisbon once, live in SF. Can confirm I loved it and want to move there.
euroderf
Is there much I.T. in Porto ? I've heard it's a great town.
callumprentice
I have a real soft spot for Lisbon and Nazare - hoping to live there one day after I retire.
Lots of similarities with the last city I lived in - San Francisco: Big red suspension bridge that spans the gateway to the ocean (same vendor), cable cars running on impossibly hilly streets, cosmopolitan, diverse LGBTQ+ friendly people, amazing food, nearby vineyards, blossoming tech scene....
mykowebhn
I lived in both San Francisco and Lisbon, and the only glaring similarities are the two bridges and the many hills. The Lisbon I love is much more charming, less money-oriented, less pretentious, and safer. The sad thing is that it's becoming much more like America. The more it tries to become like San Francisco, the sadder I become.
gusmally
>The sad thing is that it's becoming much more like America
It cannot help that more and more Americans are moving there.
SomebodySmart
One day at a beach I watched a wave crash in and saw the sand washing up in a blur. I scooped up some sand and examined it closely and it dawned on me: Close-up bitmaps of sand are a perfect source of randomness. It's easy to shuffle the sand and take another picture and by doing this repeatedly you can capture enormous random key for secure encryption. One machine does the work of composing the message and saved it onto a thumb drive. You take that out and put it into a machine that only encrypts. Then you take the cryptext onto another medium and load it into a machine that is connected to the internet and transmit. Hackers cannot reach the machine that does the work.
npodbielski
OK looks cool but otherwise can be achieved much easier with few zener diodes and amplifiers. But I know... Does not looks that cool.
srinathkrishna
I've been to Cloudflare's SF office many times for meetups and even for an interview and their lava lamp wall is one of the coolest stuff I've seen in an office.
So cool to see that they've built something similar in their Portugal office.
thiht
> It’s exciting to see waves in Portugal now playing a role in keeping the Internet secure, especially given Portugal’s deep maritime history.
I hope we should not read too much in the hanging of rainbows in Austin, Texas
nis0s
I wonder how random it is truly given that the environment something resides in shapes and changes it. For instance, imagine if there is some sort of environmental variable (temperature, humidity, light, or more) which produces an indiscernible, yet impactful, change in the behavior of the wall over time. In that way, there may not be true randomness, but a complex pattern which can be studied and acquired over time.
eastdakota
If you figure out how to model this fluid dynamics accurately over any reasonable period of time, call me. Lots and lots of more valuable things you could do with that, e.g., accurately predicting the weather.
nis0s
I was speaking more to something which happens with some consistency even in a random system. For instance, waves hitting the same piece of rocks over and over till the rocks take on a certain shape.
Reminds me of this Turkish artist Refik Anadol
https://refikanadol.com/works/bosphorus/
Bosphorus is a data sculpture inspired by high frequency radar data collections of Marmara Sea provided by Turkish State Meteorological Service in every 30 minutes intervals. The data collection of 30 days long sea surface activity transformed into a poetic experience and visualized on a 12 meters by 3 meters long LED media wall. The art work exhibited at Pilevneli Gallery on 11th of December, 2018 – 27th of January, 2019 in Istanbul, Turkey.