What people want

When searching for ideas for a new project or a startup the best strategy is to solve your own problem. A second best method is to think hard of what problems other people might have that you can solve. Let's do the second by seeing what people are searching on the internet. Services like Google Trends give a glimpse into this, however, they don't make the full list of queries available. Luckily most search engines provide a feature for suggesting search queries, based on what others have searched in the past. This data can be relatively easily fetched using an automated script. While fetching all possible queries would be too time-consuming, a specific subset can still give interesting insights.

I look at queries of the form "How can I ... ?". While asking such a direct question is not the most efficient way to use a search engine, people seem to do it all the time. Recently, Q&A sites have started exploiting this fact, by collecting possible answers on a separate page for each question. I have to admit, my original meta-idea was to build a Q&A service populated with the most frequently searched questions but this idea is probably already used by some of the Q&A sites and besides, building a SEO spam empire is not my first priority at the moment.

"How can I ..." is obviously not the only way to ask questions, to get a better picture, I should have looked at "How to...", "What if...", "Where can I ..." and possibly many others, however, this one question already seems to give a nice overview of the problems people face. So I proceeded to download the complete list from Google Suggest using a perl script, which took a few days to complete and then sorted the 61588 queries by the frequency number that Google provides.

The list is a fairly interesting read, in alphabetical order it goes from "How can I abort" to "How can I zoom out in wow". The number given by Google is probably closely related to the frequency of a query, however, it must be some approximation (the numbers are too round) and possibly other parameters are factored in (the numbers seem a bit too high). Also it seems to incorporate somehow the frequency of individual words (see later). For many of the queries it needs some thinking to figure out what people really wanted to ask. When you are really stuck with a problem, often you are unable even to ask the right questions.

CORRECTION: as pointed out by Adrian Zavelcuta and Bones in the comments below and by Mark Fulton and apollo on the HN thread, I made an error in interpreting the data. The numbers given by Google for each query are not telling how frequently a query was asked, but rather, it seems, they are related to the popularity of the words and combinations of words (from the query) in a large corpus of website data. This way they are still related to how popular these words and groups of words are, however, not as queries, but as the text of websites. According to the Google Suggest help page, the suggested queries are all real queries, i.e. they have been issued in this exact form a number of times, which in my opinion still makes the list below an interesting read, however my interpretation of the order as indicating query frequency was wrong. I thank everyone for the feedback, and apologize for the misunderstanding. I leave the text below unedited, but consider this correction as you continue.

There are a few caveats with this data:

Google say that they filter out violent, pornographic or hateful content, otherwise the query list should be comprehensive.
I am not sure about the cut-off number for the frequencies, the lowest number found was 1160, so it could be 1000. There is a long tail of diverse and interesting questions, so probably lot of gems are missed as they are below the threshold.
People who search in "how can i ..." type of questions are probably not a representative sample of the whole population. My guess is that they are less tech-savvy users on average.
There are no misspelled words, which suggests that the data has been corrected (some misspellings are so common that they should show up otherwise). Spell-checking seems to be at word level only, some of the queries make no sense as sentences.
Some queries seem to have abnormally high numbers. This could be a subtle bias due to the design of the system. I assume the queries entered while using the suggest feature are counted as well. In this case it could be that some queries are prominently suggested (for example because they are alphabetically close to a common query), so when a user sees the suggestion they think "it's not what I was searching for, but might as well click it, to see what it is..." This leads to a positive feedback loop, resulting in runaway success for some queries.
Another possible explanation is that the algorithm somehow favors "bursty" queries. Depending on how often the suggestion list is updated, it could be that some query is very frequent for a few days and Google mistakenly extrapolates this for the overall popularity of the query.
A third possible explanation is that frequencies of individual words are factored in. In this case queries with common words are featured prominently. This is almost certainly the case and explains the popularity of "How can I model for Next" and similar queries.
It was noted elsewhere that the suggested queries are somewhat tailored to local cultural sensitivities. This seems to affect just a few manually added or removed queries, probably not having much effect on the dataset as a whole. However, still worth noting that I fetched the data using a computer in Germany.
This list contains only English queries. Obviously, non-English-speakers will formulate different queries.
At a quick read, it seems there are many queries related to topical events (July 2010), popular artist, films, songs, surprisingly many consumer technology questions: a tiring amount of Facebook this, Iphone that, besides the ever popular "how can I lose/gain/maintain weight/muscles/fat" in all combinations.

Without further ado, here are the top 200 queries (scroll down for some analysis and for the full dataset):

     1	how can i be on made	23200000000
     2	how can i view a will	6770000000
     3	how can i get reason 4 for free	6500000000
     4	how can i read a will	5210000000
     5	how can i be of help	5050000000
     6	how can i make a will	4230000000
     7	how can i be of service	3610000000
     8	how can i sign in	3560000000
     9	how can i go on facebook	3550000000
    10	how can i do facebook	3540000000
    11	how can i be an aka	3160000000
    12	how can i do a blog	2920000000
    13	how can i have a blog	2870000000
    14	how can i make a site	2850000000
    15	how can i become more social	2780000000
    16	how can i search in google	2690000000
    17	how can i have a website	2580000000
    18	how can i make my windows 7 genuine	2560000000
    19	how can i search in windows 7	2480000000
    20	how can i know what i want to do	2470000000
    21	how can i become more popular	2420000000
    22	how can i model for next	2350000000
    23	how can i find new friends	2330000000
    24	how can i tell if a man is in love with me	2280000000
    25	how can i do ca	2260000000
    26	how can i do website	2250000000
    27	how can i get c form	2240000000
    28	how can i get office 2007 for free	2130000000
    29	how can i write html	2120000000
    30	how can i tell if someone opened my email	2110000000
    31	how can i work in us	2090000000
    32	how can i get there	2090000000
    33	how can i be the best	2090000000
    34	how can i become more environmentally friendly	2070000000
    35	how can i tell if he came in me	2040000000
    36	how can i assist you	2030000000
    37	how can i run older programs on windows 7	1900000000
    38	how can i print an email	1900000000
    39	how can i contact facebook	1900000000
    40	how can i be more happy	1880000000
    41	how can i make pictures	1870000000
    42	how can i find people online	1850000000
    43	how can i know who viewed my profile in facebook	1830000000
    44	how can i tell what version of office 2007 i have	1790000000
    45	how can i find out what version of windows 7 i have	1780000000
    46	how can i find my son	1780000000
    47	how can i make a change	1760000000
    48	how can i be in a film	1730000000
    49	how can i see a spirit	1690000000
    50	how can i go to facebook	1670000000
    51	how can i find people for free	1670000000
    52	how can i blog on facebook	1670000000
    53	how can i make an email	1660000000
    54	how can i work in the us	1650000000
    55	how can i get my son back	1650000000
    56	how can i check what version of flash i have	1650000000
    57	how can i know my net speed	1640000000
    58	how can i call this home	1640000000
    59	how can i do that	1620000000
    60	how can i blog	1620000000
    61	how can i make videos	1590000000
    62	how can i check if a website is up	1570000000
    63	how can i find people on facebook	1540000000
    64	how can i start share business	1530000000
    65	how can i copyright a website	1520000000
    66	how can i search twitter	1510000000
    67	how can i like school	1500000000
    68	how can i tell how tall my son will be	1490000000
    69	how can i open an email	1480000000
    70	how can i be on time	1480000000
    71	how can i get a email	1470000000
    72	how can i fit in	1470000000
    73	how can i email facebook	1470000000
    74	how can i have money	1460000000
    75	how can i do a website	1460000000
    76	how can i change office 2007 product key	1460000000
    77	how can i workout at home	1440000000
    78	how can i get what i want	1440000000
    79	how can i search facebook	1430000000
    80	how can i get the new facebook 2010	1430000000
    81	how can i email videos	1430000000
    82	how can i help people	1420000000
    83	how can i copy a page	1420000000
    84	how can i check which version of windows i have	1420000000
    85	how can i see someones facebook pictures	1390000000
    86	how can i email the view	1380000000
    87	how can i view a street	1370000000
    88	how can i get money at 13	1370000000
    89	how can i get in contact with facebook	1370000000
    90	how can i be in a movie	1360000000
    91	how can i be good	1360000000
    92	how can i find who i am related to	1350000000
    93	how can i use would	1340000000
    94	how can i get my hearing back	1340000000
    95	how can i find what i want to do	1340000000
    96	how can i find out how many copies of a book have been sold	1340000000
    97	how can i email a video	1340000000
    98	how can i create a game	1340000000
    99	how can i work in usa	1310000000
   100	how can i get free windows 7	1310000000
   101	how can i get better at being on top	1310000000
   102	how can i open a website	1300000000
   103	how can i sell my house quickly	1290000000
   104	how can i get a free grant	1260000000
   105	how can i get through to you	1250000000
   106	how can i open a word 2007 file	1240000000
   107	how can i be better with money	1240000000
   108	how can i update my twitter from facebook	1230000000
   109	how can i find employer identification number	1230000000
   110	how can i work in the states	1220000000
   111	how can i pay online	1220000000
   112	how can i know someones facebook password	1220000000
   113	how can i get into university	1220000000
   114	how can i buy a home	1220000000
   115	how can i tell he is in love with me	1210000000
   116	how can i get my son to read	1210000000
   117	how can i find an employer identification number	1200000000
   118	how can i use facebook mobile	1190000000
   119	how can i start a blog	1190000000
   120	how can i post a blog	1190000000
   121	how can i find my sign	1190000000
   122	how can i check a company	1190000000
   123	how can i upgrade to windows 7 for free	1180000000
   124	how can i not love you download	1180000000
   125	how can i make a dvd	1180000000
   126	how can i find out what i am good at	1180000000
   127	how can i get in to politics	1170000000
   128	how can i get free items	1170000000
   129	how can i add twitter to facebook	1170000000
   130	how can i work with google	1150000000
   131	how can i create my email account	1150000000
   132	how can i see who is online on facebook	1140000000
   133	how can i get an email	1140000000
   134	how can i create a blog	1140000000
   135	how can i check what version of windows i have	1140000000
   136	how can i become more green	1140000000
   137	how can i get my old number back	1130000000
   138	how can i tell if i am add	1120000000
   139	how can i make a game	1120000000
   140	how can i find people on yahoo	1120000000
   141	how can i find out who i am related to	1120000000
   142	how can i use windows 7	1110000000
   143	how can i use twitter	1110000000
   144	how can i know that i am in love	1110000000
   145	how can i get a house for free	1110000000
   146	how can i tell if an email has been received	1100000000
   147	how can i make a video with pictures	1100000000
   148	how can i contact yahoo	1100000000
   149	how can i start a book	1090000000
   150	how can i find music i like	1090000000
   151	how can i make it in life	1080000000
   152	how can i find out if a house has been sold	1080000000
   153	how can i email twitter	1080000000
   154	how can i get more facebook friends	1070000000
   155	how can i send a email	1060000000
   156	how can i search for blogs	1050000000
   157	how can i get word 2007 for free	1050000000
   158	how can i open a shop	1040000000
   159	how can i find out what type of cat i have	1040000000
   160	how can i change my behaviour	1040000000
   161	how can i create a page on facebook	1030000000
   162	how can i twitter	1020000000
   163	how can i read password	1020000000
   164	how can i make new friends	1020000000
   165	how can i help the world	1020000000
   166	how can i get some money	1020000000
   167	how can i email music	1020000000
   168	how can i be more beautiful	1020000000
   169	how can i open a business	1010000000
   170	how can i go to america	1010000000
   171	how can i contact twitter	1010000000
   172	how can i tell if i am fit	1000000000
   173	how can i start my own website for free	1000000000
   174	how can i get on the internet for free	1000000000
   175	how can i contact facebook by email	1000000000
   176	how can i check if an email has been read	1000000000
   177	how can i be down	1000000000
   178	how can i use facebook	999000000
   179	how can i create an rss feed	984000000
   180	how can i get fans on facebook	979000000
   181	how can i get more turned on	975000000
   182	how can i dream about what i want	974000000
   183	how can i be good on top	971000000
   184	how can i check net speed	970000000
   185	how can i get high at home	966000000
   186	how can i open a company	965000000
   187	how can i get to facebook	965000000
   188	how can i live in the us	962000000
   189	how can i even try to go on	961000000
   190	how can i tell how long a house has been on the market	960000000
   191	how can i find out someones facebook password	958000000
   192	how can i contact google	957000000
   193	how can i create an icon	956000000
   194	how can i be content	953000000
   195	how can i tell who searches for me on facebook	950000000
   196	how can i find out who links to me	948000000
   197	how can i be an extra	947000000
   198	how can i tell a girl that i like her	944000000
   199	how can i find people on the internet	943000000
   200	how can i find out if there is a will	943000000

Queries seem to fall into a few general categories, many queries meaning essentially the same. Let's look at a few such groups, especially with regards to our original goal of finding project ideas.

     1	how can i be on made	23200000000

First I thought this was a mistake, but it seems to be the name of a TV show. Probably the first due to the use of common words (see above).

     2	how can i view a will	6770000000
     4	how can i read a will	5210000000
     6	how can i make a will	4230000000
   200	how can i find out if there is a will	943000000

It seems people think a lot about wills. Also it might be just because it is a frequent word.

    11	how can i be an aka	3160000000

Apparently some sorority. Also, common acronym.

    22	how can i model for next	2350000000

Apparently a modelling agency, same as before.

    25	how can i do ca	2260000000
    27	how can i get c form	2240000000

Not sure what these are.

    31	how can i work in us	2090000000
    54	how can i work in the us	1650000000
    99	how can i work in usa	1310000000

A common question.

    34	how can i become more environmentally friendly	2070000000

A valid question and nice that people ask this. Maybe not enough reliable info available.

    48	how can i be in a film	1730000000
    90	how can i be in a movie	1360000000

Oh, the vanity :)

    49	how can i see a spirit	1690000000

     3	how can i get reason 4 for free	6500000000

Apparently some music software. Still strange that it is at number 3.

     8	how can i sign in	3560000000

Sign in where? this could be from really non-technical users who are presented with the "sign in" prompt somewhere and they don't understand what is required.

     9	how can i go on facebook	3550000000
    10	how can i do facebook	3540000000
    12	how can i do a blog	2920000000
    13	how can i have a blog	2870000000
    14	how can i make a site	2850000000
    17	how can i have a website	2580000000
    26	how can i do website	2250000000
    29	how can i write html	2120000000
    50	how can i go to facebook	1670000000
    52	how can i blog on facebook	1670000000
    53	how can i make an email	1660000000
    60	how can i blog	1620000000
    69	how can i open an email	1480000000
    71	how can i get a email	1470000000
    75	how can i do a website	1460000000

These suggest that many people are still left out of technology because they find services too complicated. Need for even simpler blogging, site creation, etc. platforms?

    18	how can i make my windows 7 genuine	2560000000

Probably from users who see some warning about "genuine" whatever and don't understand what it means.

    28	how can i get office 2007 for free	2130000000
    65	how can i copyright a website	1520000000
    76	how can i change office 2007 product key	1460000000
    80	how can i get the new facebook 2010	1430000000
    83	how can i copy a page	1420000000
    86	how can i email the view	1380000000
    87	how can i view a street	1370000000
   100	how can i get free windows 7	1310000000

??? :)

    19	how can i search in windows 7	2480000000
    30	how can i tell if someone opened my email	2110000000
    37	how can i run older programs on windows 7	1900000000
    38	how can i print an email	1900000000
    56	how can i check what version of flash i have	1650000000
    57	how can i know my net speed	1640000000
    62	how can i check if a website is up	1570000000
    66	how can i search twitter	1510000000
    79	how can i search facebook	1430000000
    81	how can i email videos	1430000000
    84	how can i check which version of windows i have	1420000000
    97	how can i email a video	1340000000

Mostly valid questions.

    39	how can i contact facebook	1900000000
    73	how can i email facebook	1470000000
    89	how can i get in contact with facebook	1370000000

What do you want to tell them ?

    44	how can i tell what version of office 2007 i have	1790000000
    45	how can i find out what version of windows 7 i have	1780000000

2007 and 7 I guess, am I missing something ?

    43	how can i know who viewed my profile in facebook	1830000000
    85	how can i see someones facebook pictures	1390000000

Not again.

    16	how can i search in google	2690000000

This is just too funny.

     5	how can i be of help	5050000000
     7	how can i be of service	3610000000
    82	how can i help people	1420000000
    91	how can i be good	1360000000

These are quite nice.

    15	how can i become more social	2780000000
    21	how can i become more popular	2420000000
    23	how can i find new friends	2330000000
    24	how can i tell if a man is in love with me	2280000000
    55	how can i get my son back	1650000000

Not too surprising.

    20	how can i know what i want to do	2470000000
    33	how can i be the best	2090000000
    40	how can i be more happy	1880000000
    67	how can i like school	1500000000
    70	how can i be on time	1480000000
    74	how can i have money	1460000000
    78	how can i get what i want	1440000000
    95	how can i find what i want to do	1340000000

Don't we all want the same.

    72	how can i fit in	1470000000

Don't do it, not worth it.

    35	how can i tell if he came in me	2040000000
    36	how can i assist you	2030000000
    64	how can i start share business	1530000000
    68	how can i tell how tall my son will be	1490000000

    32	how can i get there	2090000000
    41	how can i make pictures	1870000000
    47	how can i make a change	1760000000
    59	how can i do that	1620000000
    61	how can i make videos	1590000000

Be more specific.

    42	how can i find people online	1850000000
    46	how can i find my son	1780000000
    51	how can i find people for free	1670000000
    63	how can i find people on facebook	1540000000

Seems there is indeed a need that social networks try to satisfy.

    58	how can i call this home	1640000000

Name of some song ?

    77	how can i workout at home	1440000000

TV commercials have been trying to explain this for the past 40 years.

    88	how can i get money at 13	1370000000
    92	how can i find who i am related to	1350000000
    93	how can i use would	1340000000
    94	how can i get my hearing back	1340000000
    96	how can i find out how many copies of a book have been sold	1340000000
    98	how can i create a game	1340000000

Mostly good questions.

The full list can be found here: in alphabetical order, sorted by frequency (a bit more than 1000 pages).

The script used: fetch.pl

My other projects at Lkozma.net

© 2010 László Kozma (Lkozma@gmail.com)