Social and search based content creation

by on 30th November 2016

For the past few months, I’ve been talking a lot about the idea of using common, freely available tools to automate or simplify the common tasks faced by SEOs and content producers. A little while ago, I wrote about how you can use Xpath and text analysis to create pitch content. In that vein, I thought I’d bring something this time using a similar idea for creating content for your own site.

Gathering Inspiration

The first part of creating content is, of course, finding ideas for what to create. In the case of content for one’s own site, that means either something designed to address a specific business need (and we’ll look at this in a post some other time), or content designed to attract relevant, useful traffic. That’s what we’ll be looking at here.

On my own site, I write infrequently and broadly, on a variety of things. However, the focus is generally one of three things: writing, programming or the creative industries. These are subjects that are broadly covered though, so finding something unique to say can be a challenge. With that in mind, here’s the process I use when I’m going to write a post, with a real example.

Our initial data comes from Twitter. There’s a hashtag used for asking question of literary agents, which is #askagent. All well and good for getting a subject people are talking on, but there’s a lot of spam/low quality cruft in there too. Thinning it down manually could be time consuming. Fortunately, there’s a couple of simple things we can do to make life easier.

Firstly, we’re going to restrict the tweets shown to those that contain questions. Secondly, we’ll exclude anything that doesn’t have replies. That gives us a new query of ? filter:replies, which produces far better results. We can then use the following xpath on that page to extract all the tweets:

Note that you can use the or operator, (the pipe) to say “mathces this xpath or that one”

You’ll need to do a little bit of clean-up after (to get tweets on one line if there’s line breaks, and to replace the “/” you’ll see before usernames with “@”), but what you should end up with is something that looks like this…

@labellaraquella@PFDAgents #askagent does being a mix of writer/performer put you off? (Asking for a friend ahem)
@XenaKnox@PFDAgents #askagent if a book you’re trying to sell doesn’t, do you work with the author on another project or part ways?
@mk_golding@laurabirdland #askagent if I have written one book but the next one is right at the top of agent’s wish list, should I mention in query?
@laurabirdland@XenaKnox @PFDAgents we’re just sitting here going…”is it lame to say we just really love books?” #askagent
@XenaKnox@PFDAgents #askagent what do you love about your job? What do you ‘hate’ about your job?

Five minutes with this gave me 200 tweets and users. Now we can start to do some interesting things…

Finding Influencers

Because these are all questions around a hashtag, we can use that to find influencers. If we take our list of tweets, paste them into a regex parser and use this simple regex:

…we can then get a list of every user mentioned. So go to that site, put in the tweets above and pick “List” on the tools bar at the bottom. We can take that list, put it into Excel and use a countif to see how many times every user there was mentioned. That gave me the following list:

  • @mushenska: 36
  • @LZats: 20
  • @PFDAgents: 13
  • @JuliaChurchill: 13
  • @TheWorkConf: 10

That gives me a quick list of people who are influential in the space. I can then follow them to see what they tweet about and get a gauge of what these people want to see by what content they share. We can also scan all these tweets with the content analysis tool I mentioned last time. Now we start finding common phrases and words, like:

  • “what do you”
  • “how do you “
  • “what is your”
  • “do you have”
  • “query”
  • “novel”
  • “book”
  • “story”
  • “letter”
  • “response”

It’s pretty easy to then go back through the tweets, now filtered for those words and phrases, to see what the most common types of tweets are. For example, the novel tweets I pulled are:

  • @PFDAgents How long do you think it takes for a good novel to grow/be developed? #askagent
  • @mushenska Hello. Do you always read some of the novel, or dismiss on the query letter and/or synopsis alone? #askagent
  • @mushenska I keep telling my clients to try to keep their first-time novels to about 90k words (not 70k or 120k)… am I wrong? #askagent
  • @mushenska if pitching a novel and it’s the first in a series, should you mention other books that follow? (Thanks for doing #askagent )
  • @mushenska For literary fiction what’s the current ‘desirable’ length for novel by new/unpublished writers? #askagent
  • @mushenska calling back in a MS? 5 agents have the full of previous novel, but no response after months/prompts? WIP much better! #askagent
  • @mushenska last question. What’s a normal expected advance for a debut genre novel sold to a “big” publisher? (Ballpark). #askagent
  • @mushenska This year, I’ve gotten 4 short stories published and written a collection’s worth. Query re: that or 1st write a novel? #askagent
  • @soumeya_b @ECarrieHowland What is the word count minimum for a novel or story collection to be viable to publishers? Thanks! #askagent
  • @PSLiterary @carlywatters What are you currently looking for in YA novels? What are you NOT looking for? #askagent
  • @laurenspieller #askagent in multiple pov novel, is it a good idea to draft the query letter from the antagonist’s point of view?
  • @LZats Have a Bachelor’s in film focused on screenwriting. Worth including in a query bio for a novel or not relevant? #askagent
  • @LZats How do you feel about using non-book comps for a novel? If comps aren’t listed, is that a red flag? #askagent
  • @LZats How do you feel about short stories that are turned into novels? Is it a hard sell? Good thing for new authors to do? #askagent
  • @HellieOgden @Viking_Ma @sallyanne_s @pinkbeltauthor How do you feel about a novel aimed at adults but with a teenage protagonist? #AskAgent
  • @DElliottPrize Should you compare your novel to others in same genre and if so, how many and how recent? Thanks! #AskAgent #DiscoverADebut

These are all great questions, and the beauty of it is that because we’ve said “only give me tweets with replies”, we can pick the best of these, see what the answers were, and then turn them into useful content.

The Search Angle

So we’ve got a list of potential things to write about. Now let’s look and see if there’s search volume around those topics. For this, turn to your favourite keyword research tool, and see what comes back. If you want, you could even run adwords campaigns on these phrases, to get really accurate data. Whatever floats your boat. Once you’ve whittled your topics down to purely useful subjects, now you can take that and create content around it.

With the content then produced, you can reach out to the people who asked the question and present them with your content. You can also market that content yourself, in whatever form you see fit.

Taking this further, because generally you’re going to get a lot of content out of this, you can use it to flesh out a content calendar when you’re thin on ideas. Also, because you’re creating series’ of content around a theme, you can combine your output together into guides, downloads, podcasts/video blogs etc.

If you’ve any thoughts, questions, or ideas for taking this further, please leave a comment below!


  1. Hey Pete
    This is quite an unusual post. I must say, I never thought of generating ideas from tweets in such a organized way. Randomly, I would get an idea or two but the way you have explained, it is quite a unique idea to come up with some great writing ideas. Thanks for sharing. (55 words)

Comments are closed.

Get insights straight to your inbox

Stay one step ahead of the competition with our monthly Inner Circle email full of resources, industry developments and opinions from around the web.