This has been attempted many times. They all fail the same way.
These general data models start to become useful and interesting at around a trillion edges, give or take an order of magnitude. A mature graph model would be at least a few orders of magnitude larger, even if you aggressively curated what went into it. This is a simple consequence of the cardinality of the different kinds of entities that are included in most useful models.
No system described in open source can get anywhere close to even the base case of a trillion edges. They will suffer serious scaling and performance issues long before they get to that point. It is a famously non-trivial computer science problem and much of the serious R&D was not done in public historically.
This is why you only see toy or narrowly focused graph data models instead of a giant graph of All The Things. It would be cool to have something like this but that entails some hardcore deep tech R&D.
There are open source projects moving toward this scale, the GraphBLAS for example uses an algebraic formulation over compressed sparse matrix representations for graphs that is designed to be portable across many architectures, including cuda. It would be nice if companies like nivida could get more behind our efforts, as our main bottleneck is development hardware access.
To plug my project, I've wrapped the SuiteSparse GraphBLAS library in a postgres extension [1] that fluidly blends algebraic graph theory with the relational model, the main flow is to use sql to structure complex queries for starting points, and then use the graphblas to flow through the graph to the endpoints, then joining back to tables to get the relevant metadata. On cheap hetzner hardware (amd epyc 64 core) we've achieved 7 billion edges per second BFS over the largest graphs in the suitesparse collection (~10B edges). With our cuda support we hope to push that kind of performance into graphs with trillions of edges.
>These general data models start to become useful and interesting at around a trillion edges
That is a wild claim. Perhaps for some very specific definition of "useful and interesting"? This dataset is already interesting (hard to say whether it's useful) at a much tinier scale.
It was a widely observed heuristic going back to the days when the Semantic Web was trendy. The underlying reason is also obvious once stated.
Almost every non-trivial graph data model about the world is a graph of human relationships in the population. If not directly then by proxy. Population scale human relationship graphs commonly pencil out at roughly 1T edges, a function of the population size. It is also typically the highest cardinality entity. Even the purpose isn’t a human relationship graph, they all tend to have one tacitly embedded with the scale implied.
If you restrict the set of human entities, you either end up with big holes in the graph or it is a graph that is not generally interesting (like one limited to company employees).
The OP was talking about generalizing this to a graph of people, places, events, and organizations, which always has this property.
It is similar to the phenomenon that a vast number of seemingly unrelated statistics are almost perfectly correlated with GDP.
This is not a "general purpose data model", though. A better example would be Wikidata which at about 100M nodes and 1B edges (so orders of magnitude less than that 1T claim) is already enabling plenty of useful queries about all sorts of publicly-available data and entities.
The graph database market has a deserved reputation for carefully crafting scaling claims that are so narrowly qualified as to be inapplicable to anything real. If you aren't deep into the tech you'll likely miss it in the press releases. It is an industry-wide problem, I'm not trying to single out Neo4j here.
Using this press release as an example, if you pay attention to the details you'll notice that this graph has an anomalously low degree. That is, the graph is very weakly connected, lots of nodes and barely any edges. Typical graph data models have much higher connectivity than this. For example, the classic Graph500 benchmark uses an average degree of 16 to measure scale-out performance.
So why did they nerf the graph connectivity? One of the most fundamental challenges in scaling graphs is optimally cutting them into shards. Unlike most data models, no matter how you cut up the graph some edges will always span multiple shards, which becomes a nasty consistency problem in scale-out systems. Scaling this becomes exponentially harder the more highly connected the graph. So basically, they defined away the problem that makes graphs difficult to scale. They used a graph so weakly connected that they could kinda sorta make it work on a thousand(!) machines even though it is not representative of most real-world graph data models.
> It is a famously non-trivial computer science problem and much of the serious R&D was not done in public historically.
Could you point us to any public research on this issue? Or the history of the proprietary research? Just the names might help - maybe there are news articles, it's a section in someone's book, etc.
Either or both could be an interesting starting point for something like that. I tried to find something for the BBC and was surprised they didn’t have anything. I would have figured public media would have been a great resource for this.
Given 6 degrees is rooted in reality, this means we can draw causal graphs from anyone (bad) to anyone (we don't like) and then invent specious reasons why it means "it's all connected, man"
That said, some networks of shorter paths than 6 are interesting. Right now, there's a 1:1 direct path from these documents to a bunch of people with an interest in confounding what evidentiary value they have in justice processes. That's more interesting to me, than what the documents say right now.
Microsoft Access form that connects via IIS to an Excel spreadsheet acting as a database. Also the server it's running on is sitting on a wooden table.
I think you meant one shudders. And yeah, Snowden made it clear there's orders of magnitude more data than this graph explorer for them to sift through.
Internet search engines have their origins in government projects fwiw. They had search engines before Alta Vista, used for searching data sets that pre-date the internet, and some of the people involved in those went to work on the original commercial search engines.
As "The Rest Is Politics" podcasts points out, the meagre consequences mostly came to Brits: Ghislaine Maxwell, Prince Andrew aka Andrew Mountbatten, and the former UK embassador to the US.
Americans don't really do accountability all that much. People there who get to face the consequences are usually the ones that significantly harmed the financial interests of the very rich. Madoff, Holmes, Bankman. They operate more on a vengance than accountability system.
If the evidence is strong enough, sure. But as much as I like a "the elites are pedophiles" witchhunt, given that the Biden administration sat on it, it's probably safe to conclude the evidence isn't great. The Trump administration is trying to get another wack at it, but given their recent history of investigations, it's probably safe to conclude that's purely politically motivated than some cold case that got cracked.
It's a bit too bad that the network visualisation relies on d3: it is really slow with big networks, and the force directed algorithm is far from the best.
Have you tried using JS libraries built specifically to visualise graph networks such as Sigma.js, Vivagraph or Cytoscape?
Shameless plug: if OP is looking to stay on d3, he could also try slotting in my C++/WASM versions[1] of the main d3 many-body forces. Not the best, but I've found >3x speedup using these for periplus.app :)
This is great work to show relationship and connections. The government gets scared from these types of efforts as there are many members who are extremely guilty of crimes related to this and others.
We need to expand on network mapping with data and areas as well.
>A force-directed graph is a technique for visualizing networks where nodes are treated like physical objects with forces acting between them to create a stable arrangement. Attractive forces (like springs) pull connected nodes together, while repulsive forces (like electric charges) push all nodes apart, resulting in a layout where connected nodes are closer and unconnected nodes are more separated
I think it would be better and faster if the website calculated the positions of the nodes in the background (with a good enough limit of iterations), and then showed the result. Animating 4k nodes and 25k edges (15k by default) is a waste of CPU and is laggy even on my high-end CPU. But maybe the author was limited by the tools used.
Yes, the dataset also has three entries for Virginia Giuffre, "Virginia L. Giuffre", "Virginia Roberts Giuffre", and "Jane Doe Number 3 (Virginia Roberts)"
LLMs are awful for this. I've got a project that's doing structured extraction and half the work is deduplication.
I didn't go down the route of LLMs for the clean up, as you're getting into scale and context issues with larger datasets.
I got into semantic similarity networks for this use case. You can do efficient pairwise matching with Annoy, set a cutoff threshold, and your isolated subgraphs are merger candidates.
I wrapped up my code in a little library if you're into this sort of thing.
I read a recent observation that people subject to discovery are often making purposeful typos in key names in order for the communication to remain under the radar.
If you look at this graph and your prescient thought is "haha take that MAGA" then you are a brainwashed ideologue. This graph gives a window into the layers of rot in our political system. The complexity is perfectly represented by its form but it seems like your graph is just a big arrow that says "orange man bad".
After seen this I interested in a map of each person to assist with knowing who they are, who they worked for during the email date, and who they currently work for.
Given how strongly he was against it, this is pretty clearly a ploy. He could release them unilaterally if he hadn't just reopened the investigation which he himself shut down.
This obviously the correct lens but note that the 2008 plea deal was so neutered by the time of settlement it made it somewhat easy to stay friends with him.
This is of course ontop of the 2006 Florida prostitution charge though.
Especially when Epstein was paying off journalists at the NYT and intimidating other outlets.
But point being those people that were friends with him had to know. Whether it was socially acceptable by the elite because the public wasn't aware isn't very relevant.
I read the gp as saying you should just check the sources, not defending.
I mean here's a weird example. Searching Donald Trump there's the headline
(1994-06 Wexner Mansion NYC)
Donald Trump forced to perform oral sex and physically abused 13-year-old female plaintiff and 12-year-old female.
Like that sounds weird... DT forced to rape? That doesn't make sense to me. The longer summary reads
A declaration from Tiffany Doe (pseudonym) testifying that she witnessed Jeffrey Epstein and Donald Trump sexually abuse a 13-year-old girl and other minors during parties from 1990-2000 in New York City.
It references House Oversight 025937. The actual document looks much more like that summary. Here's a snippet
7. It was at these series of parties that I personally witnessed the Plaintiff being forced to perform various sexual acts with Donald J. Trump and Mr. Epstein. Both Mr. Trump and Mr. Epstein were advised that she was 13 years old.
It gets worse so if you want to look further it's Case 1:16-cv-04642 Document 1-2 Filed 06/20/16 Page 1 of 2.
So far the paragraph summaries seem to be accurate in my poking around but the headlines are mixing ordering and have other weird errors like this. Anyways, always good to check when things are as serious as this...
It’s wild to see people who fell for the pee tapes still around almost a decade later insisting that, this time, the latest accusations will be supported by evidence rather than Glenn Beck style dot connecting.
I admit I fell for Russiagate. I even voted for Biden in 2020. But I learned my lesson. What’s your excuse? Does Trump’s view on immigration upset you so much you’re willing to continue trusting a media that has done nothing but lie about him for a decade?
As far as I can tell, what fundamentally differentiates people who simply don’t like Trump for all the legitimate reasons to dislike Trump from the people who go full blown Rachel Maddow is deep-seated liberal universalism. For most people, Trump is merely a shady character and a serial liar. That’s why my dad hates Trump. But for others, he attacks the core of their worldview. Those folks will gobble up any shred of innuendo no matter how far-fetched and no matter how discredited the source.
Yes it is. Trump cut ties because Epstein was poaching employees. That was confirmed by the employee in question, Virginia Giuffre, who always maintained that she was a normal masseuse at Mar-a-Lago: https://abcnews.go.com/US/virginia-giuffre-trump-jeffrey-eps...
What I don't understand is the pretense of defending Trump at all. I mean, it's clear that even if you watched Trump assault a 13 year old with your own eyes, it wouldn't impact your support for him. Why pretend that there is some moral divide between Bill Clinton and Donald Trump in this when you can just say "I support Donald Trump no matter what, and despite Bill Clinton no matter what"?
Personally I've never been shocked that some of the most powerful people in the world like to go to a private sex-island where they could do as they pleased. That's precisely the incentive to becoming so incredibly powerful in the first place: to be able to pursue personal gain with increasingly less consequences.
Clinton at least has not been in office for 25 years. Trump is still in office. Surely the priority should be to get the bad people out of institutions asap...?
Pre-2009 records on Trump there are nasty. One example:
> ... It was at these series of parties that I personally witnessed the Plaintiff being forced to perform various sexual acts with Donald J. Trump and Mr. Epstein. Both Mr. Trump and Mr. Epstein were advised that she was 13 years old. I personally witnessed four sexual encounters that the Plaintiff was forced to have with Mr. Trump during this period, including the fourth of these encounters where Mr. Trump forcibly raped her despite her pleas to stop.
Only difference between Clinton and Trump is that Trump is still president.
That's a lie that has already been proven false since Trump's entire trip was documented.
Love how we have actual evidence against people but discussions always devolve into some conspiracy related to Trump.
----
Based on the available evidence, there is no confirmed meeting between Trump and Epstein in 2017. While both men were in Palm Beach during Thanksgiving week 2017, there is no direct evidence they met.
Here's what we know about their presence in Palm Beach that week:
- Trump was at Mar-a-Lago from November 21-26, 2017
- Epstein owned a mansion in Palm Beach and was known to be in the area
- Epstein mentioned both Trump and himself being "down there" (Palm Beach) in an email exchange on November 23, 2017
While there were claims circulating online that Trump spent Thanksgiving with Epstein in 2017, these claims have been thoroughly investigated and found to be unsubstantiated
Trump's official calendar for that week shows his activities included:
- Thanking military members on a virtual call
- Visiting Coast Guard members at Lake Worth Inlet Station
- Playing golf with Tiger Woods and Dustin Johnson
Even if Trump cut off ties with Epstein in 2017, he should clearly be held accountable for his past actions. Here's 2 pretty damning emails:
---
Epstein to Maxwell 2011-04-02
> i want you to realize that that dog that hasn’t barked is trump... [VICTIM] spent hours at my house with him ,, he has never once been mentioned. police chief. etc. im 75% there
---
Epstein to Ruemmler 2018-08-23
> you see, i know how dirty donald is. my guess is that non lawyers ny biz people have no idea. what it means to have your fixer flip
Not really. After Epstein got convicted in 2008, he set about trying to rehabilitate his image, to be seen as a philanthropist, a patron of science, and (perversely) a supporter of women and girls. He hired reputation management consultants to help carry out the project, with one of the models they used being Mike Milken (of Drexel infamy), who ultimately secured a pardon from Trump. A lot of prominent people, knowingly or not, served as "useful idiots" in this project, often due to financial incentives that were not wholly selfish. For example, the MIT and Harvard scientists whose labs and research he funded, and who visited his island for science-themed retreats. Clinton was probably another of Epstein's useful idiots, being lured in through his Clinton Global Initiative and the promise that Epstein, with his ample wealth, could help greatly expand it.
Just like here you could get a timeline of key events, a graph of connected entities, links to original documents.
Newsrooms might already do this internally idk.
This code might work as a foundation. I love that it's RDF.
reply