Support syndication category

In case you didn't already see...

2011-03-14T22:31:34Z

Paid Account Trials cannot create new syndicated accounts. This is intentional, and not a bug. If a user on a Paid Account Trial asks about creating a syndicated account, you can gently nudge them to purchase the real thing.

New FAQ and new stock!

2009-12-05T22:26:14Z

Hello gang! I have come to bring Holiday tidings and joy and these tidings are in the form of:

FAQ 306 (userdoc post) and

stock answer 1287.

These were both added to address a new source of requests for us, people going "OMG GOOGLE READER HAS ALL MY PROTECTED ENTRIES" and acting like they've uncovered a terrible secret that we're in cahoots with Google to pass off private entries (we're not. at least not until their check clears).

So, if a request comes in where someone is concerned about a 3rd-party service or about RSS exposing their protected entries in general, you can point them to the new FAQ and apply the new stock as appropriate.

In other news, do any of you have edits you'd like to see to the stocks/to the FAQs/is there anything I can do to make you being in syn more fun (or to get you involved in syn, if you're not already)? Please comment to this request (I've screened comments) and I will try to make your wish come true for Christmas.

Love and bunnehs,
~gergles

A new syn priv!

2009-03-03T06:04:56Z

Effective immediately, all syn supporthelps will also receive the canview:entryprops priv as part of their priv package. This priv will permit you to view meta-information about entries by going to this page and pasting in an entry link.

Syn-wise, the useful lines are mostly:

User Date/Time: 2009-02-20 05:00:00
Server Date/Time: 2009-02-20 05:22:49

# syn_id: http://xkcd.com/546/
Unique id of syndication item

# syn_link: http://xkcd.com/546/
Original URL of syndication item

User date/time is the date and time that were listed in the feed, and server date/time is the time synsuck ran and posted the entry to LJ. syn_id is the GUID or other unique ID that LiveJournal picked up for the entry, and syn_link is the <link> attribute's information. For an outstanding guide to reading entryprops output, check this post to support_interim.

Of course, you're being given the priv to do syn stuff, there may be sensitive information exposed through entryprops, abuse of the tool will you get whomped, etc. etc.

Thanks for all you all do for the category!

The Bizarro Feed, or, When 404 Doesn't Mean 'Not Found'

2009-01-13T01:23:07Z

(This post was migrated from the syn_training community, originally made by seaofdestiny.)

This was a strange request.

As the user mentioned, the profile page for zakhad_blog mentioned that the last fetch resulted in an error '404 Not Found' -- but if you clicked on the 'XML' button on the profile, you could see content in your browser.

After a bit of digging, including a command-line utility (I used lwp-request, which comes with Perl's LWP package), I found something weird: when I fetched the feed, I got data, yes. But when I added some debugging flags to show me the HTTP status, it told me "404 Not Found"! A bit more digging with netcat and speaking raw HTTP to the server (always fun ^^~) corroborated that.

So it appears that the server, for whatever reason, was delivering data, but with a status code indicating that there was no data there! My browser, apparently, figured that if the content was not text/html (likely an error page) but text/xml, then there was indeed content despite the status code; but LiveJournal, apparently, chose to trust the status code (as I think it should have).

Fortunately, the user was able to provide an alternate feed location which behaved properly (returning '200 OK' as its HTTP status), and I used syn_editurl to change the feed URL to the new URL.

In related news, if anyone is interested in a quick tutorial in using lwp-request and/or netcat to diagnose syn-related problems, let me know. If there's interest, I'll try to make time to write something down about what I do.

(I use a Unix shell for this, but Perl runs on Windows, too, and I wouldn't be surprised if netcat had been ported to Windows as well, so the information might be applicable even if you don't have access to a Unix shell but 'only' a Windows command line.)

Translated accounts

2009-01-09T23:52:44Z

(This post was migrated from the syn_training community, originally made by seaofdestiny. Please excuse the friends page mess!)

This support request reminded me of something which I learned and which might come in handy.

The user is talking about 'translation', and it appears that 'syndication' is what we'd call it.

This is probably due to the Russian terminology in this area; 'syndication' appears to be translated as 'трансляция' (translyatsiya) in Russian, and a 'syndicated account' as 'трансляционный аккаунт' (translyatsionnyj akkaunt) or 'транслируемый аккаунт' (transliruyemyj akkaunt). So this is standard Russian terminology on LiveJournal (don't know about on the wider Internet).

My Russian is limited to recognizing a word here and there so I don't know whether the words are related to the English words 'translation/transliteration', but occasionally you'll see a support request from a Russian speaker mentioning that they want to 'translate their blog to LiveJournal' or create a 'translated account'. Now you'll know what they're likely to mean with that.

New Policy: "Too Big" feeds

2009-01-03T01:47:19Z

Happy New Year, synners!

For Syndicated Accounts that aren't updating because of the "Too Big" error, we (the admins) now have a tool available to us that lets us flag the feed to update anyway, as long as it is less than 500 KB in size. Therefore, please don't say "Sorry, the limit's 300K, [FAQ]" for feeds that are between 300 and 500 KB - just leave them for an admin to fix. If you want to be super-helpful, IC with the feed file size.

If the feed is more than 500 KB in size, feel free to say "the FAQ says they can only be so big" and to handle them as before.

If you have questions about the policy, please e-mail us at support@.

Cool? Cool!

Brad asks, "How many points would be good?"

2003-11-03T06:15:34Z

In this thread on the News post about userpics, bradfitz asks:

We never meant for those to be restrictive. Just for anti-abuse. (we're always paranoid with new features)
The syn subsystem is pretty damn robust/fast now, though, so maybe we should up everything.
How many syn points you think would be good?

Just so y'know. I've given my 2c. :-)

A stroll through synsuck

2003-08-12T19:30:56Z

What follows below is an overview and a line-by-line walkthrough of the part of the LiveJournal source code which handles the retrieval of RSS feeds, conversion to journal entries, and storage of those entries in the LiveJournal database.

Understanding this document is not required of Syndication volunteers. I expect that most volunteers will not use this, or maybe will find the overview interesting. It's a reference I wrote up to help me diagnose one problem a long time ago, and I figured I might as well make it public.

(That said, I encourage anyone to comment with questions, requests for clarifications, and so forth, even if you think your question is silly.)

The current version of synsuck.pl can be obtained by clicking on the highest version number in its entry in the CVS repository browser:

http://cvs.livejournal.org/browse.cgi/livejournal/bin/maint/synsuck.pl

Walkthrough of `bin/maint/synsuck.pl` v1.25

Last modified: 2003-08-12 02:25 UTC

Overview

For each syndicated account:

request RSS file
if response is larger than 150KB, exit and try again in 60 minutes
check for misidentified character set and repair
parse RSS
take most recent 20 items of feed in reverse order
delete any entries older than two weeks (max friends page time) from LJ database
check to see if feed uses different <link>s in each entry; if so, pull up the last set of <link>s from the database for later comparison
For each item in a feed:
- Check to see if this exact item (all fields) has been seen before; if so, skip to next item
- Store fields from XML into fields of an LJ entry
- Check if the <link> already exists in the database
- If so, perform 'editevent' to edit entry; otherwise, perform 'postevent' to post new entry
Update account name, website, bio with global title, URL, description from RSS feed
Decide when to poll the feed next
Update count of number of users reading this syndicated account

Detail

Lines 4-9:

loads required Perl libraries, declares global variables.

11:

The synsuck subroutine is entered into the hash which contains all of LiveJournal's maintenance tasks. This subroutine and the others in that hash are alled from bin/ljmaint.pl, which is launched as a periodic job via cron.

20:

We retrieve all users whose statusvis is 'V' (syndicated) and whose 'checknext' field is in the past (ie, where checknext has come up since the last run).

25-end (312):

This is the main loop of synsuck.pl, which executes once for every syndicated account returned from the previous query (l. 20).

27-33:

Create an anonymous subroutine which sets the 'lastcheck' field in the 'syndicated' table to the current time, and the 'checknext' field in that table to the current time plus N minutes where N is supplied as an argument, and records the status provided as an argument, for the syndicated user being looked at. Note that this subroutine is not called yet, only created, and a reference to it is stored (in $delay) for later use. Below, this subroutine will be referred to as the scheduler.

37-47:

Requests the feed URL with libwww-perl, including HTTP headers to request the document only if it has been modified since the last request and if the entity tag has changed since the last request. (This way, per HTTP 1.1 we get new data unless both conditions are true -- i.e., we increase our chances of getting new data from a recalcitrant caching proxy along the way.)

We check the size of the response on the way via a subroutine that is called on the incoming data. If the response is larger than 150*1024 bytes we discard it immediately and set the $too_big flag.

48:

If $too_big is set, we tell the scheduler to check the feed again in 60 minutes and store "toobig" as the status, and go on to the next feed.

50-55:

If we received HTTP 304 (Not Modified), we call the scheduler subroutine with a status of 'notmodified' and a delay of 60 minutes if the feed has readers, or 1 day if the feed has no readers.

57-66:

Comment justifying the subsequent block that deserves reproduction here:

 # WARNING: blatant XML spec violation ahead... 
 # 
 # Blogger doesn't produce valid XML, since they don't handle encodings
 # correctly.  So if we see they have no encoding (which is UTF-8
 # implictly) but it's not valid UTF-8, say it's Windows-1252, which
 # won't cause XML::Parser to barf... but there will probably be some
 # bogus characters. [...]

67-73:

Code referred to by above comment, which looks for "<?xml" and "encoding=" and stores the value of "encoding"; if not present and data does not appear to be UTF-8 (via LJ::is_utf8()), edits the content in-place to contain "encoding='windows-1252'".

75-83:

"Another hack" which checks for Windows smart quotes in a document which claims to be UTF-8; if present, edits the existing "encoding=" string to read "encoding='windows-1252'".

85-89:

Parse XML with XML::RSS::parse(), trapping exceptions, and storing the resulting RSS object.

90-99:

If an exception is raised, call the scheduler subroutine with the status "parseerror" and a delay of three hours, clean up the error returned by XML::RSS::parse(), and store as the user's "rssparseerror" userprop, then go to the next feed.

102:

Check that the "items" method of the RSS object returns an array (of items in the feed). If it doesn't, call the scheduler subroutine with a status of "noitems" and a delay of 3 hours, then go to the next feed.

104:

Store a copy of the items in the RSS feed in the reverse order that they appear in the feed.

107:

Take only the 20 bottommost items from the feed (i.e., the first 20 items in our stored copy).

111-116:

Connect to the LJ database. If connection fails, call the scheduler subroutine with status "nodb" and delay of 15 minutes, then go to the next feed.

118-130:

Retrieve ids of all articles in the database for this account user which are older than MAX_FRIENDS_VIEW_AGE (default 2 weeks) and delete them from the database.

132-143:

Try to determine if the <link> field of each item can be expected to be unique; for each item that has a link field, store the link field, and if we've stored more than one link field, assume that links can be expected to be unique.

149-158:

If the links can be expected to be unique, pull all of the links of already-stored entries from the database, and store those links with their entry IDs.

163-249:

Iterates over each item in the feed to possibly store it:

169-171:

Remove Perl's internal UTF-8 flag from the title, link and description fields of the item, so that Perl does not know it is UTF-8 even if it is. This prevents Perl from performing UTF-8 conversions for us even when we're doing it ourselves. These automatic conversions are what introduced the recent double-encoding bug where one syndicated entry would break UTF-8 encoding for a whole friends page.

173-178:

Take an MD5 hash (via LJ::md5_struct) of all of the fields and values in the item. Check to see if that digest has already been stored in the synitem table for this syndicated account; if it has, assume we've already seen this item and go on to the next item. If it hasn't, store the MD5 hash and account ID in the synitem table.

180:

Now that we believe this is a new item, increment a counter, $newcount (which is specific to this account).

182-183:

Remove whitespace from the beginning and end of the <description> field.

186-190:

If the item has a <link> field, store it in an <a href> in a temporary variable, $htmllink.

192-209:

Set the fields of the LJ "postevent". Of interest:

'Subject' is <title>
'Event' is $htmllink prepended to <description>
The 'syn_link' property is <link>

211-214:

If the <description> contains <p> or <br>, assume that the contents are preformatted, and set the "don't autoformat" property.

216-235:

If this <link> appears in the list (from l.149) of existing <link>s for the feed, decrement $newcount, and prepare to perform an "editevent" instead of a "postevent". Maintain the original posting time.

237-239:

Perform the postevent or editevent, using the current time as post time.

240-244:

If successful, ensure that one second elapses between the time the post was posted and now, otherwise wait one second. This prevents two articles from being posted at the same time, so the order in which they appear in the friends list matches the order they appeared in the feed.

245-249:

If the postevent or editevent failed, set $errorflag.

[Repeat from 163 for each item.]

245-255:

If any postevent or editevent for this feed failed (ie, $errorflag is set), call the scheduler subroutine with a status of "posterror" and a delay of 30 minutes, and go to the next feed.

257-288:

Update syndicated account's userinfo:

258:

Load 'url' and 'urlname' property from existing userinfo.

260-264:

If the RSS feed has a <title>, replace the syndicated account's name with the feed <title> and set the "urlname" user property to the feed <title>.

266-269:

If the RSS feed (not an individual item) has a <link> set the "url" user property to the <link>.

271-287:

If the RSS feed (not an individual item) has a <description>, set the user's bio to the <description>, otherwise record that the user has no bio.

293-297:

Decide when to poll the feed next. And I quote:

     # FIXME: this is super lame.  (use hints in RSS file!)

If there were new articles this time, interval is 30 minutes and status is "ok"; otherwise interval is 60 minutes and status is "nonew". Call the scheduler subroutine with the appropriate values.

299-304:

Update reader count if there were new articles, or if it has never been updated, but otherwise leave it at current value.

307:

If there are no readers, use the scheduler subroutine to change next poll interval to 1 day.

309-311:

Store last-modified date, entity tag, status, and interval to next poll in database.