../ RSS Feed Generation with Emacs-Lisp

Currently I manage my youtube channels with a single youtube.org file, where each channel has its own file.

The org file contains all the useful metadata for each video. Specifically, for each video I have the following data

A list of properties
- title
- id generated by youtube
- playlist
- pubblication date
- private folder
- public foler
Three specific sections
- description
- timestamps
- references

Below you can see a screenshot of the file that showcases the data associated to a specific video.

Now there's a lot of material to cover with respect to how I handle this data using emacs, emacs-lisp and org-mode. For example, the public and private written into the properties are used to generate a public github repository associated to the channel. You can find these repositories at the following URLs

Recently I've also added an RSS feed for both of my channels that contains the lists of videos and that is updated anytime I publish a new video. Below you can find the URLs for the RSS feeds

In this post I want to explain how I generate such feed with some emacs-lisp code, as I find it a simple yet practical application of the language, and it showcases the integration between it and Emacs.

In the post I will explain the code using a top-down approach. If you're interested in just reading the code however, you can find it in the following URL:

https://blog.leonardotamiano.xyz/files/rss-feed-emacs-lisp/rss.el

#Some Elisp A Day Keeps The Boring Tasks Away

We start with the basic data structure that I use to represent my two youtube channel. For each channel I have a specific repository.

(setq yt2git/metadata-file "youtube.org")

(setq yt2git/repos
      '(

	((:name . "yt-it")
	 (:label . "IT")
	 (:private . "/home/leo/projects/FOUNDATIONS/yt-it")
	 (:public . "/home/leo/projects/PUBLIC/yt-it"))

	((:name . "yt-en")
	 (:label . "EN")
	 (:private . "/home/leo/projects/FOUNDATIONS/yt-en")
	 (:public . "/home/leo/projects/PUBLIC/yt-en"))

	))

To access specific values within this structure we can use the yt2git/get-prop function.

(defun yt2git/get-prop (field repo)
  (assoc-default field repo))

Within the specific context of RSS generation, we're interested in two main functions, which are

yt2git/gen-rss, generates a new RSS feed with all videos present within the youtube.org file.
yt2git/update-rss, updates an already existing RSS feed by adding the latest videos added to the youtube.org file.

These functions are actually simple wrappers marked with interactive, which exposes them to the functions list obtained when using M-x within emacs.

(defun yt2git/gen-rss ()
  (interactive)
  (yt2git/gen-rss-by-repo (yt2git/select-repository))
  )

(defun yt2git/update-rss ()
  (interactive)
  (yt2git/update-rss-by-repo (yt2git/select-repository))
  )

The yt2git/select-repository is used to interactively choose a repository. Notice how I'm using the ivy library to deal with user input.

(defun yt2git/select-repository ()
  (let* ((repo-name (ivy-read "Project: "
			      (mapcar
			       (lambda (repo) (assoc-default :name repo))
			       yt2git/repos)))
	 (repository (car (seq-filter
			   (lambda (e) (string= (assoc-default :name e) repo-name))
			   yt2git/repos))))
    repository))

#Generation of RSS

Let us now consider the generation of an initial RSS feed. This is implemented by the function yt2git/gen-rss-by-repo, which takes a repository as input.

(defun yt2git/gen-rss-by-repo (repo)
  (let* ((public (yt2git/get-prop :public repo))
	 (rss-feed (concat public "/feed/rss.xml")))
    (with-temp-buffer

      ;; initial tags
      (insert
       (concat "<rss version=\"2.0\" xmlns:atom=\"http://www.w3.org/2005/Atom\">\n\n"
	       "<channel>\n"
	       (format "<title>Leonardo Tamiano - Youtube %s </title>\n" (yt2git/get-prop :label repo))
	       "<link> https://www.youtube.com/@LT123/videos </link>\n"
	       "<description> Aggiornamenti e metadati sui video caricati </description>\n"
	       (format "<atom:link href=\"https://leonardotamiano.xyz/rss/%s/.xml\" rel=\"self\" type=\"application/rss+xml\" />\n" (assoc-default :name repo))
	       "\n"))

      ;; one for each video
      (mapcar
       (lambda (video) (yt2git/gen-rss-entry repo video))
       (yt2git/videos repo))

      ;; closing tags
      (insert "</channel>\n")
      (insert "</rss>")

      (write-file rss-feed)
      )
    ))

The code uses the with-temp-buffer to open a temporary buffer that is used to create the initial parts of the RSS feed. We put at the top some initial tags. The result of this is shown below.

<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">

<channel>
<title>Leonardo Tamiano - Youtube EN </title>
<link> https://www.youtube.com/@LT123/videos </link>
<description> Aggiornamenti e metadati sui video caricati </description>
<atom:link href="https://leonardotamiano.xyz/rss/yt-en.xml" rel="self" type="application/rss+xml" />

Then we do a mapcar, which iterates over all the elements returned by the yt2git/videos function. This function returns a list of elements, where each element contains the metadata for a specific video. To go from the metadata of the single video to the RSS entry for that particular video we use the function yt2git/gen-rss-entry. Once the temporary buffer has been filled up, we write the entire content on the buffer into the rss-feed variable, which is found in the public folder associated to that repository.

Consider now the implementation for the yt2git/gen-rss-entry function.

(defun yt2git/gen-rss-entry (repo video)
  (let* ((delim "\n\n-------------------------\n\n")
	 (description
	  (with-temp-buffer
	    (insert (yt2git/gen-description-by-video repo video))
	    (beginning-of-buffer)
	    (replace-string "\n" "<br>\n")
	    (beginning-of-buffer)
	    (replace-regexp "\\(https://.*\\)<br>" "<a href=\"\\1\"> \\1 </a> <br>")
	    (beginning-of-buffer)
	    (replace-regexp "\\(http://.*\\)<br>" "<a href=\"\\1\"> \\1 </a> <br>")
	    (buffer-string))))
    (insert
     (concat
      "<item>\n"
      "<title>" (assoc-default :title video) "</title>\n"
      "<guid>" (format "https://leonardotamiano.xyz/rss/%s.xml#" (assoc-default :name repo)) (assoc-default :youtube video) "</guid>\n"
      "<link>" (format "https://youtu.be/%s" (assoc-default :youtube video)) "</link>\n"
      "<pubDate>" (yt2git/format-rss-date (assoc-default :pubblication video)) "</pubDate>\n"
      "<description>"
      (format "<![CDATA[\n%s\n]]>" description)
      "</description>\n"
      "</item>\n\n"
      ))

    )
  )

Notice how in the computation of the description we open another temporary buffer and we call the function yt2git/gen-description-by-video. The rest of the function simply fills up the XML template with all the interesting data with the elisp functions that deal with strings, between concant and insert. Notice also the usage of the function yt2git/format-rss-date to format the date of the video to be compliant with the RSS standard.

An example of such entry is shown below.

<item>
<title>Getting used to BurpSuite</title>
<guid>https://leonardotamiano.xyz/rss/yt-en.xml#mdYKh4PkMCo</guid>
<link>https://youtu.be/mdYKh4PkMCo</link>
<pubDate>dom, 18 feb 2024 00:00:00 GMT</pubDate>
<description><![CDATA[
Hi, and welcome to this new video!<br>
<br>
In this video I discuss the BurpSuite community edition tool, one of the most used tools when it comes to Web Exploitation.<br>
<br>
If you're interested in more videos of the series please let me know!<br>
<br>
-------------------------<br>
<br>
TIMESTAMP<br>
<br>
00:00 Introduction<br>
01:20 Initial Setup<br>
13:25 Installing PortSwigger CA certificate<br>
17:40 Starting the web application<br>
18:30 Configuring the scope<br>
24:50 Proxy interception<br>
28:55 Repeater<br>
33:40 Decoder<br>
36:00 Comparer<br>
37:00 Analyzing cookie structure<br>
42:00 Intruder<br>
45:56 Sequencer<br>
47:00 Dashboard<br>
48:50 Extensions<br>
51:00 Conclusion<br>
<br>
-------------------------<br>
<br>
REFERENCES<br>
<br>
- Material: <a href="https://github.com/LeonardoE95/yt-en/tree/main/src/2024-02-18-web-exploitation-burpsuite"> https://github.com/LeonardoE95/yt-en/tree/main/src/2024-02-18-web-exploitation-burpsuite </a> <br>
- SecureBank: <a href="https://github.com/ssrdio/SecureBank"> https://github.com/ssrdio/SecureBank </a> <br>
- BurpSuite Community Edition: <a href="https://portswigger.net/burp/communitydownload"> https://portswigger.net/burp/communitydownload </a> <br>
   <br>
<br>
-------------------------<br>
<br>
CONTACTS<br>
<br>
- Blog: <a href="https://blog.leonardotamiano.xyz/"> https://blog.leonardotamiano.xyz/ </a> <br>
- Github: <a href="https://github.com/LeonardoE95?tab=repositories"> https://github.com/LeonardoE95?tab=repositories </a> <br>
- Support: <a href="https://www.paypal.com/donate/?hosted_button_id=T49GUPRXALYTQ"> https://www.paypal.com/donate/?hosted_button_id=T49GUPRXALYTQ </a> <br>

]]></description>
</item>

The generation of the date is pretty simple.

(defun yt2git/format-rss-date (custom-date)
  (format-time-string "%a, %d %b %Y 00:00:00 GMT" (date-to-time custom-date))
  )

The yt2git/gen-description-by-video function is probably the most complex ones of the bunch, because we actually have to read the org file, extract the three sections that represent description, timestamps and references, and combine everything in a single string.

;;
;; With this function we can generate a custom description for a
;; specific video.
;;
(defun yt2git/gen-description-by-video (repo video)
  (let* ((org-file (concat (assoc-default :private repo) "/" yt2git/metadata-file))
	 (query (concat (yt2git/label2txt :youtube) "=\"" (assoc-default :youtube video) "\""))
	 (labels (if (string= (assoc-default :label repo) "IT")
		     (list '(:ref . "RIFERIMENTI") '(:contacts . "CONTATTI"))
		   (list '(:ref . "REFERENCES") '(:contacts . "CONTACTS"))))

	 ;; First we extract the raw data from the file. To do this we
	 ;; first select the outline based on the YOUTUBE-ID, and then
	 ;; we iterate over all entries of that outline.
	 ;;
	 (raw-data (car (org-map-entries
			 (lambda ()
			   (org-map-entries
			    (lambda () `(,(org-get-heading) . ,(org-get-entry)))
			    t 'tree))
			 query
			 (list org-file))))
	 ;;
	 ;; Then we extract description. timestamp and references from
	 ;; the various outlines.
	 ;;
	 ;; NOTE: for this to work properly we need to have this exact
	 ;; order when writing the org file. That is, first we write
	 ;; description, then timestamp and finally the references.
	 ;;
	 (raw-description (cdadr raw-data))
	 (raw-timestamp (cdaddr raw-data))
	 (raw-references (cdr (cadddr raw-data)))
	 (template (format (concat
			    "%s"
			    "\n\n-------------------------\n\n"
			    "TIMESTAMP"
			    "\n\n"
			    "%s"
			    "\n\n-------------------------\n\n"
			    ;; "REFERENCES"
			    (assoc-default :ref labels)
			    "\n\n"
			    "- Material: https://github.com/LeonardoE95/%s/tree/main/src/%s-%s\n"
			    "%s"
			    "\n\n-------------------------\n\n"
			    ;; "CONTACTS"
			    (assoc-default :contacts labels)
			    "\n\n"
			    "- Blog: https://blog.leonardotamiano.xyz/\n"
			    "- Github: https://github.com/LeonardoE95?tab=repositories\n"
			    "- Support: https://www.paypal.com/donate/?hosted_button_id=T49GUPRXALYTQ\n"
			    )
			   raw-description
			   raw-timestamp
			   (assoc-default :name repo)
			   (assoc-default :pubblication video)
			   (assoc-default :public video)
			   raw-references))
	 (full-description (with-temp-buffer
			     (insert template)
			     (beginning-of-buffer)
			     (delete-matching-lines "begin_example")
			     (delete-matching-lines "end_example")
			     (buffer-string)
			     )))
    (kill-new full-description)
    full-description
    ))

Finally, the last function we need to discuss is the yt2git/videos function, which takes a repository as input and returns the list containing the metadata for each video. The core ingredient of the function is the call to org-map-entries, which reads all the data from the youtube.org file.

(defun yt2git/videos (repo)
  (let ((org-file (concat (assoc-default :private repo) "/" yt2git/metadata-file))
	(query (concat (yt2git/label2txt :youtube) "={.+}")))
    (with-temp-buffer
      ;;
      ;; read youtube.org and extract over all org-mode outlines and select
      ;; only those that have a YOUTUBE property assigned. From the
      ;; org-entries only extract the things we care about to construct our
      ;; ToC, which are the following values
      ;;
      ;; Title, Playlist Pubblication Date, Video
      ;;
      (setq-local entries (org-map-entries
			   (lambda ()
			     (let ((props (org-entry-properties)))
			       (list `(:title        . ,(assoc-default (yt2git/label2txt :title) props))
				     `(:playlist     . ,(assoc-default (yt2git/label2txt :playlist) props))
				     `(:pubblication . ,(assoc-default (yt2git/label2txt :pubblication) props))
				     `(:youtube      . ,(assoc-default (yt2git/label2txt :youtube) props))
				     `(:private      . ,(assoc-default (yt2git/label2txt :private) props))
				     `(:public       . ,(assoc-default (yt2git/label2txt :public) props)))
			       ))
			   query
			   (list org-file)
			   ))

      ;; do not show all entries that have not yet received a
      ;; pubblication date or that have not been published yet.
      ;;
      (setq-local entries (seq-filter
			   (lambda (e)
			     (and
			      (not (string= (assoc-default :tbd e)
					    (assoc-default :pubblication e)))
			      (org-string<= (assoc-default :pubblication e)
					    (format-time-string "%Y-%m-%d"))))
			   entries))

      ;; sorts using the pubblication date, ordering them from most
      ;; recent video to the oldest one. Notice that while sort performs
      ;; an in-place sort, the reverse function instead creates a new
      ;; list and returns the reversed list.
      (setq-local entries (reverse
			   (sort entries (lambda (a b)
					   (string< (assoc-default :pubblication a)
						    (assoc-default :pubblication b)
						    )))))
      entries
      )
    )
  )

Here the yt2git/label2txt is used to centralize where I have written the hard-coded string that represent the various properties.

(defun yt2git/label2txt (label)
  (assoc-default
   label
   '(
     (:title        . "TITLE")
     (:playlist     . "PLAYLIST")
     (:pubblication . "PUBBLICATION")
     (:youtube      . "YOUTUBE")
     (:public       . "PUBLIC")
     (:private      . "PRIVATE")
     (:tbd          . "TBD")
     ))
  )

#Update of RSS

The update of the RSS feed, implemented in yt2git/update-rss-by-repo, is much simpler than the generation, for obvious reasons. We just need to locate the place within the feed where we need to start write the new entries, and then for each entry we generate the RSS text using once again the yt2git/gen-rss-entry previously shown and add the text to the feed. At the end we overwrite the initial file.

(defun yt2git/update-rss-by-repo (repo)
  (let* ((public (yt2git/get-prop :public repo))
	 (rss-feed-filename (concat public "/feed/rss.xml"))
	 (rss-feed-content (with-temp-buffer
			     (insert-file-contents rss-feed-filename)
			     (buffer-string)
			     ))

	 ;; get to the insertion point for new videos (after </channel>\n\n)
	 (insertion-point (with-temp-buffer
			    (insert rss-feed-content)
			    (beginning-of-buffer)
			    (search-forward "/>\n\n")
			    (point)
			    ))

	 ;; extract latest pubDate as a reference point
	 (latest-video-date
	  (yt2git/format-custom-date
	   (with-temp-buffer
	     (insert rss-feed-content)
	     (beginning-of-buffer)
	     (search-forward "<pubDate>")
	     (buffer-substring (point) (+ (point) 29)))))

	 ;; get all videos published after that pubDate
	 (new-videos (yt2git/videos-by-date repo latest-video-date))
	 )

    ;; for each video add a new item
    (with-temp-buffer
      (insert-file-contents rss-feed-filename)
      (goto-char insertion-point)
      (mapcar
       (lambda (video) (yt2git/gen-rss-entry repo video))
       new-videos)
      (write-file rss-feed-filename)
      )
    )
  )

Notice that here I do not call the yt2git/videos function, but rather I call the yt2git/videos-by-date. While the former gives me the metadata for all the videos made so far, the latter only gives the video with a pubblication date later than the date given as input. The implementation is actually the same as the yt2git/videos one, I just add an extra condition when filtering the entries.

;; only returns videos that are returned AFTER the given date.
(defun yt2git/videos-by-date (repo date)
  ;; ...
  ;; do not show all entries that have not yet received a
  ;; pubblication date or that have not been published yet.
  ;;
  (setq-local entries (seq-filter
		       (lambda (e)
			 (and
			  (not (string= (assoc-default :tbd e)
					(assoc-default :pubblication e)))
			  (org-string<= (assoc-default :pubblication e)
					(format-time-string "%Y-%m-%d"))
			  ;; THIS IS THE ONLY THING WE'VE ADDED IN THIS SPECIAL VERSION.
			  (org-string> (assoc-default :pubblication e)
				       date)
			  )
			 )
		       entries))

  ;; ...
  )

So, that's it! I hope it was an interesting read that showcases with a practical example the potential contained within the Emacs editor. In the future I should clean this code and also extend it with more functionality, but right now it works, as it allows me to update the RSS feed with little to no effort, and so I'll keep it as it is.